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(57) Abstract: A device and method for the automated 
storage and retrieval of trays (901a) holding subject mat- 
ter. A plurality of trays (901a) are inserted into an access 
device. A computer system is programmed to control a 
storage gantry to move the trays (901a) between the ac- 
cess device, a storage rack and a work cell gantry. 
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WO 03/064047 PCT/US03/02107 
Automated Storage and Retrieval Device and Method 

The present invention relates to automated storage and retrieval devices and methods 
for using, and in particular to such devices and methods used in conjunction with 
inspection devices for sequentially inspecting microscopic crystals. 

BACKGROUND OF THE INVENTION 
The determination of the three dimensional atomic structure of matter is one of the 
most important areas of pure and applied research. One way in which the three 
dimensional atomic structure of matter can be determined is through X-ray 
crystallography. X-ray crystallography utilizes the diffraction of X-rays from crystals 
in order to determine the precise arrangement of atoms within the crystal. The result 
may reveal the atomic structure of substances such as metal alloys, deoxyribonucleic 
acid (DNA), or the structure of proteins. 

There are very important benefits to knowing the accurate molecular structure of a 
protein crystal. For example, once the molecular structure is known, a drug designer 
can more effectively develop effective therapeutic agents and drugs. However, 
despite its promises, X-ray crystallography is limited by the fact that it is very 
difficult to grow successful crystals. 

Prior Art Method of Growing Crystals 
Protein crystals are commonly grown in the wells of micro-well plates. A micro-well 
plate is also known as a micro-titer plate or a microplate. Micro-well plates typically 
come with either 24, 48, 96, 384 or 1536 wells. A 96-well micro-well plate is shown 
in detail in FIG. 2. There are a variety of methods in which protein crystals may be 
grown. Five common ways are summarized below. 



Hanging Drop Method 
One of the main techniques available for growing crystals, known as the hanging-drop 
or vapor diffusion method, is a method wherein a drop of a solution containing 
protein is applied to a glass cover slip and placed upside down in an apparatus such as 
a vapor diffusion chamber where conditions lead to supersaturation in the protein drop 
and the initiation of precipitation of the protein crystal. 
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Sitting Drop Method 

Another method is the sitting drop method where the drop sits in a small well adjacent 
the growing solution instead of hanging over it This method provides a more stable 
drop and location. 

Aqueous Drop in Oil Method 
Another method is the aqueous drop in oil method. The drop is placed in a micro- 
well and is covered with an oil based solution. The drop stays at the bottom of the 
well as the crystal grows. 

Dialysis Method 

In another method referred to as the dialysis method (also called microbatch 
crystallization), the protein solution is contained within a semi-permeable size 
exclusion membrane and then placed in a solution of fixed pH and precipitant 
concentration. As the precipitant diffuses through the membrane into the protein 
compartment, the solubility of the protein is reduced and crystals may form. 

Gel Crystal Growth Method 
This method involves the placement of a gel into the end of small diameter glass 
capillaries. After the solutions have gelled, a protein solution is placed into one end 
(top) of the capillary and the other end is submerged in a solution of precipitating 
agent If the conditions are appropriately selected, crystal growth occurs at a point in 
the gel where the protein and precipitating agent reach the proper concentrations as 
the solutions slowly mix by diffusion. Since this is a diffusion limited process, it thus 
only occurs after an extended period of time. Crystals however, grown by this method 
are often larger and of higher quality. 

Regardless of the method chosen, protein crystal growth is a very delicate and time- 
consuming process. It can take several days to several months before crystals of 
sufficient size and quality are grown and ready for x-ray crystallography. The current 
minimum size that is typically stated is a crystal of at least 50 microns thick by 100 
microns in extent The protein crystal growing environmental conditions need to be 
rigorously maintained, from the chemistry, to the surrounding air humidity and 
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temperature, cleanliness to prevent contamination, and even lifting conditions. A 
protein crystallographer working with unknown protein families may only be about 
5% successful in growing proper sized quality crystals. With this success rate, for 
example, a 96-well micro-well plate may only have 5 wells in which good crystals are 
growing. 

Prior Art Inspection of Crystal Growth 
Currently, a laboratory technician, or operator, aided by a microscope and a 
laboratory notebook manually inspects crystals grown in micro-well plates. To 
inspect a micro-well plate, a laboratory technician dons a clean-room gown suit and 
enters a cold room in which the crystals are growing. The technician then puts a 
micro-well plate underneath the microscope and examines each well in the micro-well 
plate until all of the wells in the micro-well plate have been inspected. The technician 
then makes a mental judgement as to how he shall classify (also known as "score") 
the crystal. For example, the technician may feel that he is observing an image that 
j shows "grainy precipitation" or "ugly precipitation". Or, he may feel that the image 
shows "no crystal growth". The technician then records the classification into a 
laboratory notebook. 

The above system is riddled with opportunities for human error. An operator, 
manually inspecting a 96-well micro-well plate will take approximately 5 to 20 
minutes depending on the skill of the operator and the number of wells that contain 
interesting features, microcrystals, or crystals. The operator may be subject to 
physical fatigue, suffer eyestrain, and may be uncomfortably cold in the temperature 
controlled and generally high humidity room. The operator can be tired and confused 
and can easily make errors in manually recording data in the notebook. For example, 
the operator may observe crystal growth at well H5 (FIG. 2), but incorrectly record in 
the notebook that the crystal growth was at well H6. Additional transcription errors 
may occur when the data is transferred to a computer database. 

Research efforts are underway to try to solve the above problem, but they are 
inadequate for the needs of the industry. One such effort is described in Jurisica et al. 
"Intelligent Decision Support for Protein Crystal Growth" IBM systems Journal, Vol. 
40, No 2, 2001. Another such effort is described at the Website www.dsitech.com. 
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Current Problems with Micro-well plate Storage and Retrieval Procedures 
Typically, after a technician has inspected a micro-well plate for crystal growth, the 
micro-well plate is stored until it is time to inspect it again. The growing of protein 
crystals in micro-well plates and the accompanying inspection of the micro-well 
plates for successful crystal growth are procedures that are typically carried out 
concurrently in large quantities in laboratories. For example, a typical lab at any given 
moment may have literally thousands of micro-well plates in which protein crystals 
are attempting to grow. The growth cycle of a protein crystal can be approximately 6 
months. During the 6 month time period, a micro-well plate may be inspected up to 
approximately 12 times. If there are thousands of micro-well plates that require 
inspection, it can be a very time consuming task to manually move the micro-well 
plate from its storage location, place it under a microscope, record the results, and 
then move it back to its appropriate storage location. Moreover, there is tremendous 
opportunity for a technician to forget where a particular micro-well plate belongs. Or, 
a technician handling such a large quantity of micro-well plates can easily drop or 
otherwise damage the micro-well plates he is handling. 

What is needed is a better device and method for storing and retrieving trays 
containing micro-well plates. 

SUMMARY OF THE INVENTION 
The present invention provides a device and method for the automated storage and 
retrieval of trays holding subject matter. A plurality of trays is inserted into an access 
device. A computer system is programmed to control a storage gantry to move the 
trays between the access device, a storage rack and a work cell gantry. The computer 
system is also programmed to control the work cell gantry to move the subject matter 
to and from an automated receiving machine. In a preferred embodiment, the subject 
matter in the trays is a plurality of micro-well plates in which microscopic crystals 
may be growing and the automated receiving machine is configured to inspect and 
classify microscopic crystals. The inspecting and classifying device has an indexing 
device for sequentially placing microscopic crystals in camera-view of a camera and a 
control computer is programmed to control the indexing device and to cause the 
camera to take images of the microscopic crystals and then transfer the images, to a 
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classifying processor where the images are classified. In a preferred embodiment, the 
microscopic crystals are protein crystals that have been grown in the wells of micro- 
well plates. 

BRIEF DESCRIPTION OF THE DRAWINGS 
FIG. 1 shows a preferred embodiment of the present invention. 
FIG. 2 shows a micro-well plate. 
FIG. 3 shows a top view of the fixture plate. 

FIGS. 4 and 5 show top views of micro-well plates on the fixture plate. 

FIG. 6 shows a block diagram of a preferred embodiment of the present invention. 

FIG. 7 shows a preferred monitor. 

FIGS. 8-10 and 18-25 show steps in the sequence of operations of a preferred 

embodiment of the present invention. 

FIG. 1 1 shows hanging drops of liquid in a micro-well plate. 

FIG. 12 shows an example of aqueous drop in oil protein crystallization. 

FIG. 13 shows a top view of a micro-well plate on the fixture plate. 

FIG. 14 shows a side view of the light source shining upwards onto a micro-well 

plate. 

FIG. 15 shows a magnified view of two wells of a micro-well plate, wherein each 
well has a drop of liquid. 

FIGS. 16 and 17 show a detail view of the drops of liquid shown in FIG. 15. 

FIG. 26 shows a preferred monitor screen after a run has been completed. 

FIGS. 27 and 28 show details of other preferred monitor screens. 

FIG. 29 shows a hanging drop of liquid with crystal growth. 

FIG. 30 shows a preferred embodiment of the present invention. 

FIG. 31 shows a flowchart of an auto-focus subroutine of the present invention. 

FIG 32. shows a flowchart of a focus value subroutine. 

FIG. 33 shows a flowchart of the auto score and classify subroutine. 

FIGS. 34a - 34d show flowcharts of the classify subroutine. 

FIG. 35a -35b show the sub-classification of the crystal class. 

FIG. 36 shows the main program flow. 

FIG. 37 illustrates a side view illustrating dual filters in the light path. 

FIG. 38 illustrates a top view of the drive mechanism for the rotatable linear polarized 

filter. 
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FIG. 39 illustrates a top view of a second filter wheel. 

FIG. 40 shows the connectivity of another preferred embodiment 

FIGS. 41 A and 41B shows another preferred embodiment of the present invention. 

FIG. 42 shows the top view of a preferred tray. 

FIG. 43 shows a preferred work cell area. 

FIG. 44 shows a block diagram of a preferred embodiment of the present invention. 
FIGS. 45 - 77 show a sequence of operation of a preferred embodiment of the present 
invention. 

FIG. 78 shows a perspective view of a preferred tray. 
FIG. 79 shows another view of the tray of FIG. 78. 
FIG. 80 shows another view of the tray of FIG. 78. 
FIG. 81 shows another view of the tray of FIG. 78. 
FIG. 82 shows another view of the tray of FIG. 78. 

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT 
A detailed description of a preferred embodiment of the present invention can be 
described by reference to the drawings. 

FIG. 1 shows a preferred embodiment of the present invention. Micro-well plates 
125A - 125F are placed on fixture plate 129. In a preferred embodiment, each micro- 
well plate has 96 wells. Each well has a drop of liquid in which microscopic protein 
crystals may be growing. Computer 105 automatically controls linear actuators 115, 
150 and 160. Linear actuator 115 moves fixture plate 129 along the x-axis. Linear 
actuator 150 moves moving base 154 along the y-axis, and linear actuator 160 moves 
moving plate 162 along the z-axis. n The computer coordinates the movement of the 
linear actuators to properly position cameras 155 and 135 above each well of each 
micro-well plate 125A - 125F in sequence. Preferably, cameras 155 and 135 are 
high-resolution 2/3 inch CCD cameras, with 1,300 horizontal pixel elements by 1,030 
vertical elements. Each element is preferably 6.7 microns square with a sensing area 
of 87 mm by 6.9 mm. Cameras 155 and 135 take images of each well and transmit 
the images to computer 105 where they are digitized into a image data array of 
approximately 1,296 horizontal pixels by 1,000 vertical pixels by 8 bits of gray level 
representing the intensity (0 to 255) of each pixel. These digitized images are 
automatically recorded in the database of computer 105 and can be analyzed and 
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scored by an operator via monitor 620. The digitized images may be further 
processed, analyzed, and the contents of the individual well scored by computer 105 
executing program instructions to perform calculation? on the image data array. In a 
preferred embodiment, computer 105 is connected via a communication/control line 
to a computer network. In this manner, the present invention can be controlled from a 
remote computer. Likewise, images and data can be transmitted to the remote 
computer. 

Sequence of Operation of a Preferred Embodiment 
Micro-well plates loaded onto the System 
As shown in FIG. 1, six 96-well micro-well plates 125 A - 125F have been placed 
(either by an operator or by an external loading robot) onto fixture plate 129. A 96 
well micro-well plate 125A is shown in FIG. 2. Micro-well plate 125A has wells 
labeled Al through H12 and a bar code label 220. Micro-well plate 125A is available 
from Nalge-Nunc International, with U.S. offices in Rochester, NY. Fixture plate 129 
will hold 24, 48, 96, 384, or 1536 well micro-well plates since the micro-well plate 
external width and length are fairly standard in the industry. The 96 well micro-well 
plate will be used to illustrate the present invention. 

FIG. 3 shows a top view of fixture plate 129 just prior to loading micro-well plates 
125A - 125F. Fixture plate 129 has six cutout sections 131 that are just slightly 
smaller in length and width than micro-well plates 125 A - 125F. 

FIG. 4 shows a top view of micro-well plates 125A - 125F immediately after they 
have been placed on fixture plate 129. 

After the operator has placed micro-well plates 125 A - 125F onto fixture plate 129 as 
shown in FIG, 4, he enters the command into computer 105 (FIG. 1 and FIG. 6) to 
expand detents 510 and 520 (FIG. 5). The expansion of detents 510 and 520 firmly 
secures micro-well plates 125A - 125F against plate stops 530. 
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Recording the Bar Code Information for the Micro-well plates 
Computer 105 (FIG. 1) has been programmed to accept inputs from an operator. FIG. 
7 shows a display representing micro-well plates 125A - 125F on the screen of 
monitor 620. In FIG. 7, the operator has mouse clicked on bars 622A - 622F, which 
has caused them to turn green. By clicking on bars 622A - 622F, the operator has 
selected corresponding micro-well plates 125A - 125F to "run". The operator sends 
the command to run the selected micro-well plates by clicking on run bar 623. 

In FIG. 8, the operator has given the command to run the selected micro-well plates. 
Micro-well plate 125A has been moved to a position underneath cameras 155 and 
135. By the utilization of plate sensor transmitter/receiver 186 and reflector 188, 
information is sent to computer 105 reporting that micro-well plate 125 A is in 
position underneath the cameras. Plate sensor transmitter/receiver 186 is fixed to 
support 189 and is aligned to sense whenever a micro-well plate breaks a beam of 
light emitted by transmitter/receiver 186 and reflected by a reflector 188. Reflector 
188 is mounted on support 191 on the opposite side of the linear actuator 115. The 
plate sensor transmitter/receiver 186 and reflector 188 are preferably model # E3T- 
SR13 available from Western Switch Controls of Santa Ana, CA. 

Bar code reader 175 is also mounted to support 189 and is positioned to view bar- 
code identity label 220 (FIG. 2) attached to micro-well plate 125A when it is 
positioned underneath cameras 155 and 135. Bar-code reader 175 is preferably model 
# BL601 available from Keyence Corporation of America of Newark, NJ. Bar-code 
reader 175 communicates with computer 105 via a communication line. Information 
encoded into label 220 preferably includes: the plate serial number, the plate type 
(i.e., 24-well, 48-well, 96-well, 384-weIl, or 1536-well micro-well plate), and the well 
type (i.e., square, or rounded, hanging drop, sitting drop, constrained sitting drop). 

The information from plate sensor transmitter/receiver 186 and bar-code reader 175 is 
transmitted to computer 105 and stored for later use during the camera inspection and 
information acquisition phase. 

In FIG. 9, linear actuator 115 has moved fixture plate 129 so that micro-well plate 
125B is underneath cameras 155 and 135. In a fashion similar to that described above 
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with regards to micro-well plate 125A, information is transmitted from plate sensor 
transmitter/receiver 186 and bar-code reader 175 to computer 105 and stored for later 
use during the camera inspection and information acquisition phase. 

The above described sequence continues until all micro-well plates 125 A - 125F have 
been sensed and recorded by plate sensor transmitter/receiver 186 and bar-code reader 
175. 

Then, as shown in FIG. 10, linear actuator 115 moves micro-well plate 125A so that it 
is underneath lens 165 of camera 155. Motor 130 of linear actuator 160 moves 
moving plate 162 upward and/or downward as necessary to properly focus lens 165 
on the drop of hanging liquid over well Al. Preferably, lens 165 is set at a 
predetermined zoom. 

Inspection of Crystals 
Determining the Position of the Drop of Liquid within Each Well 
An operation to inspect each well to determine the position of each hanging drop of 
liquid is performed on micro-well plate 125A after it has been moved to the position 
shown in FIG. 10. 

FIG. 11 shows a cross section side view of wells Al - El of micro-well plate 125A. 
In a preferred embodiment, an attempt has been made to grow protein crystals in the 
hanging drops in each of the wells of micro-well plate 125 A FIG. 1 1 shows hanging 
drops al, pi, %1, 51, and el. 

The preferred method for protein crystal growth is the hanging drop method. The 
hanging drop method (also known as vapor diffusion) is probably the most common 
method of protein crystal growth. As explained in the background section, a drop of 
protein solution is suspended over a reservoir containing buffer and precipitant. Water 
diffuses from the drop to the solution leaving the drop with optimal crystal growth 
conditions. 
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In FIG. 10, lens 165 of camera 155 is over well Al of micro-well plate 125A (FIG. 2, 
FIG. 11, and FIG. 13). FIG. 13 shows a top view of micro-well plate 125A positioned 
on fixture plate 129. 

FIG. 14 shows a side view of micro-well plate 125A positioned on fixture plate 129. 
Support 191 with embedded light source 194 is positioned to the side of fixture plate 
129. Light from light guide 195 is directed upward through cutout 131 (also shown in 
FIG. 3). Light guide 195 is positioned between fixture plate 129 and plate 127 such 
that both plates can move around the light guide 195 without interference. As 
explained above, fixture plate 129 has cutouts 131 (FIG. 3) that are smaller than the 
micro-well plates 125 and located under each well plate, such that light from light 
guide 195 can be projected through the well plates when they are brought into 
position for inspection. In the preferred embodiment, light source 194 is model # 
A08925 fiber-optic backlight available from Aegis Electronics of Carlsbad, CA. 

Camera 155 (FIG. 10) inspects well Al and transmits an image to computer 105 for 
digitization. As described above, camera 155 preferably (FIG. 10) inspects well Al at 
a IX magnification so that every 6.7 micron square pixel represents approximately 6.7 
square microns on the object being measured, allowing for some small geometric 
distortions caused by the lens 165. Computer 105 has been programmed to digitize 
the camera image and then by utilizing vision software determines a position within 
well Al for the drop of liquid hanging from grease seal 361. The position of the drop 
of liquid is recorded for later use onto the hard drive of computer 105 and to a 
memory location within the computer. 

In a preferred embodiment, the vision software used to determine the position of the 
drop of liquid uses a software routine algorithm called mvt_blob_find from a 
collection of image processing tools called MVTools. MVTools is available from 
Coreco Imaging, US Office in Bedford, MA. 

After recording the position of the drop of liquid hanging from grease seal 361 in well 
Al, linear actuator 115 moves fixture plate slightly to the left so that lens 165 is over 
well Bl (FIG. 13). In a fashion similar to that described for well Al, the position of 
the drop of liquid hanging from grease seal 361 in well Bl is recorded on the hard 
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drive of computer 105 and in computer memory. For example, as shown in FIG. 15, 
computer 105 will record that drop of liquid <xl is towards the upper left-hand 
quadrant of well Al. Likewise, the position of drop of liquid pi is recorded onto the 
database of computer 105 as being in the lower right-hand quadrant of well Bl. 

In this manner, positions of the drops of liquid are recorded for cells Al - H12. In 
FIG. 18, linear actuator 1 15 has moved fixture plate 129 so that well HI is under lens 
165. 

hi FIG. 19, linear actuator 115 has moved fixture plate 129 to the left and linear 
actuator 150 has moved moving base 154 slightly rearward so 'that lens 165 is over 
well A2 of micro-well plate 125A (FIG. 13). 

In a manner similar to that described above, positions of the drops of liquid are 
recorded for cells A2 - H12 (FIG. 2, FIG. 13). In FIG. 20, linear actuator 115 has 
moved fixture plate 129 to the left and linear actuator 150 has moved moving base 
154 rearward so that well H12 is under lens 165. 

After positions of the drops of liquid are recorded for cells Al - H12 for micro-well 
plate 125 A, linear actuator 115 moves fixture plate 129 and linear actuator 150 moves 
moving base 154 so that cell Al of micro-well plate 125B is underneath lens 165 
(FIG. 21). 

In a manner similar to that described above, positions of the drops of liquid are 
recorded for cells Al - H12 for each micro-well plate 125A - 125F. In FIG. 22, linear 
actuator 115 has moved fixture plate 129 and linear actuator 150 has moved moving 
base 154 so that well H12 of micro-well plate 125F is under lens 165. 

Recording the Image of the Drop of Liquid within Each Well 
An operation to inspect each hanging drop is performed at a higher magnification 
using camera 135 with its zoom lens 145 capable of magnifications of 2.5X to 10X 
corresponding approximately to digitized pixels representing 2.68 microns square ( at 
2.5X) to 0.67 microns square (at 10X). This inspection is done for the purpose of 
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detennining whether protein crystals have grown. Zoom motor 192 controls the 
degree of zoom for zoom lens 145. Using data representing the position of the drop 
of liquid within each well obtained during the inspect-well sequence, computer 105 
(FIG. 1) automatically transmits a signal to linear actuators 115 and 150 to position 
lens 145 directly over the drop of liquid within each well. For example, in FIG. 23 
lens 145 is positioned over the top of well Al of micro-well plate 125A. Using the 
positioning data earlier obtained, lens 145 is precisely positioned so that it is able to 
zoom in on drop of liquid al (FIG. 13). FIG. 16 shows a magnified view of drop of 
liquid al. In FIG. 23, motor 130 of linear actuator 160 has moved moving plate 162 
upward and/or downward as necessary to properly focus lens 165 on drop of liquid 
al. Zoom motor 192 has manipulated lens 165 to obtain the desired degree of zoom. 
Camera 135 inspects well Al and transmits a signal representing the magnified image 
of the hanging drop of liquid to computer 105. The images are stored on computer 
105 temporarily in memory for immediate analysis and on hard disk for later analysis. 

In a similar fashion, linear actuators 115, 150 and 160 and zoom motor 192 operator 
to properly position and magnify zoom lens 145 over each hanging drop of liquid to 
obtain desired focus and magnification for image data storage. For example, FIG. 17 
shows a magnified view of hanging drop of liquid pi. 

In a manner similar to that described above during the inspect-well sequence, 
magnified images of the drops of liquids (similar to those shown in FIGS. 16 and 17) 
are recorded for cells Al - H12 for micro-well plate 125A (FIG. 2, FIG. 13). In FIG. 
24, linear actuator 115 has moved fixture plate 129 and linear actuator 150 has moved 
moving base 154 so that well H12 of micro-well plate 125A is under lens 165. 

Then, the sequence is repeated for micro-well plates 125B - 125F so that magnified 
images of the hanging drops of liquids are recorded for each cell Al - H12 for micro- 
well plates 125B - 125F. In FIG. 25, the sequence has ended for micro-well plates 
125A - 125F. Linear actuator 115 has moved fixture plate 129 and linear actuator 
150 has moved moving base 154 so that well H12 of micro-well plate 125F is under 
lens 145. 
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Manual Scoring the Drop of Liquid within Each Well 
After micro-well plates 125A - 125F have been run, monitor 620 will appear as 
shown in FIG. 26. In FIG. 26, six images representing micro-well plates 125A - 
125F appear on the screen. Above each image is a message <6 Run Comp" indicating 
that image data for hanging drops of liquid has been transferred into computer 105. 
Beneath each image are buttons 710-715 marked "S". By mouse clicking on any 
button 710 - 715, the operator may manually score for successful crystal formation 
each magnified image of each hanging drop of liquid. 

For example, in FIG. 26, the operator can mouse click on button 710 to score micro- 
well plate 125 A. 

In FIG. 27, the operator has mouse clicked on the circle representing well Al of 
micro-well plate 125 A. This has caused a magnified image to be displayed of drop of 
liquid al in screen section 716. The operator has concluded that there are no crystals 
in drop of liquid al and has therefore mouse clicked on button 717 for "NO 
CRYSTAL". On the display screen, this has caused the circle representing well Al of 
micro-well plate 125A to turn red. 

In FIG. 28, the operator has mouse clicked on the circle representing well Bl of 
micro-well plate 125A. This has caused a magnified image to be displayed of drop of 
liquid pi in screen section 716. The operator has concluded that there are crystals in 
drop of liquid pi and has therefore mouse clicked on button 718 for "CRYSTAL". 
On the display screen, this has caused the circle representing well Bl of micro-well 
plate 125A to turn green. 

In a similar fashion, the above scoring propedure is repeated until all wells Al — H12 
for micro-well plates 125A - 125F have been scored as either red (NO CRYSTAL) or 
green (CRYSTAL). 

Data Utilization 

Once micro-well plates 125A -12£F have all been scored, the operator has at his easy 
disposal a database that contains the identity of each micro-well plate that was 
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inspected along with a score summarizing whether crystal formation occurred for 
each well in the micro-well plate. The automated and efficient manner in which the 
operator is able to acquire his contrasts with the prior art method of laboriously 
inspecting each well with a microscope and the handwriting the results into a 
notebook. 

For example, to score six 96-well micro-well plates utilizing the present invention 
should take approximately no more than 10 to 15 minutes. 

In. contrast, the prior art method of inspecting six 96-well micro-well plates with a 
microscope and the handwriting the results into a notebook will take approximately 
30 to 100 minutes depending on the conditions discussed in the background section in 
addition to the time required to transcribe the results into a computer database. Plus, 
as previously explained in the background section, manual inspection and scoring is 
subject to a relatively high risk of human error. 

Second Preferred Embodiment 
In a second preferred embodiment, the depth of view of camera 135 is approximately 
50 to 100 micrometers. The crystal in the drop of liquid may be larger than the depth 
of view or there may be crystals growing at various levels within the hanging drop of 
liquid, as shown in FIG. 29. Therefore, in the second preferred embodiment, lens 145 
is focused at multiple different levels 721 - 724 and a set of images are recorded at the 
different levels so that the entire crystal may be analyzed. 

Specimen Auto-Focus 
The third preferred embodiment of the present invention utilizes a specimen auto- 
focus subroutine 300 (FIG. 31). Subroutine 300 ensures that the specimen within the 
micro-well is in-focus at the desired zoom (or magnification ratio) of image lens 145. 
Utilizing the auto-focus feature, the present invention causes camera 135 to take a 
number of images defined by a Number_of_ZjSlices. Typically, there are between 5 
and 10 slices separated in the Z-axis from one another by a Z_StepJSize. The typical 
step size is 0.05mm to 0.25 mm. The slices preferably start at a Z-Axis location 
' defined by a Start_Z value, which is typically at the bottom of the cover-slip on the 
micro-well plate. During specimen auto focus initialization 302, input data 304 is 
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received. The area of interest window within the images is further defined in the 
input data 304 by a XJWindow_Center and a Y_Window_Center plus an X_Width 
and a Y_Width. Initial settings 306 for the routine 300 are the starting value of a 
counter N, a BestJFocus, a BestJZ, and a Focus JBrror. The inspection device sets 
Z_Position(N) equal to the Start_Z location and moves camera 135 there in step 308. 
An image is acquired with camera 135 and digitized as previously described and 
stored as Image (N) in step 310. A second subroutine 312 extracts a focus value F (N) 
for Image (N) and is further described in the section for discussion for FIG. 32. A test 
is made between F(N) and the Best_Focus in step 316, such that if F(N) is greater 
than BestJFocus then BestJFocus is set to F(N) and BestJZ is set to N as shown in 
step 320 and the program flow goes onto step 322, if the test condition is not met in 
step 316 then the program flow skips step 320 and goes on to step 322. In step 322, a 
test is made to determine of all of the slices have been taken as N is tested against the 
Numberj)f JZjSlices* If N is equal to Number jOfZJSlices then program flow goes 
onto step 324. If more slice images are needed, then the flow goes to step 318. In 
step 318, Z_Position(N+l ) is set to Z_Position(N) 4- Z_StepJSize and the Z-Axis is 
moved to Z_Position(N+l ) and the program flow goes on to step 314 where N is 
incremented by 1 (one). The program flow goes back to step 310 and completes the 
loop of step 310 to step 322 until all of the image slices have been taken and then 
moves onto step 324. In step 324, BestJZ is tested against its initial value, and if it 
equals its initial value (meaning no focus was found in the focus value subroutine 
312) then it is set to a default value of the Number_of_Z_Slices divided by 2 and 
FocusJBrror is set to 1 (one) in step 326 and the program flow goes onto step 328. If 
BestJZ in step 324 has a value other than its initial value then program flow goes onto 
step 328 from step 324. In step 328 a Bestjmage image is set to the image slice at 
best focus by setting Bestjmage equal to Image(Best_Z). Also, a BesHmageJZ 
value is set equal to ZJPosition(Best_Z) and the flow goes onto step 330 which is the 
RETURN part of the subroutine and program flow returns to the main software flow. 

As illustrated in FIG 32. an image(N) focus F(N) subroutine 312 is further detailed, 
starting at the step Start 401. A pointer Jo Jmage(N) 404 is provided in step 402. In 
step 408 the image(N) is convolved with a standard 3x3 Sobel_Horizontal filter 416 
to produce an Image(N)_H wherein horizontal edges within the image are 
emphasized. In step 410 the image(N) is convolved with a standard 3x3 
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Sobel_Vertical filter 418 to produce an Image(N)_V wherein vertical edges within 
the image are emphasized. In step 414, both the horizontal edge emphasized image 
Image(N)_H and the vertical.edge emphasized image Image(N)_V are summed pixel 
by pixel, during the summing process any resulting negative pixel values are set to 
zero and any resulting pixel values that are greater than 255 are set to 255, to produce 
an image Image(N)_Focus. In step 420, a simple variance F(N) is calculated for the 
pixels within a window of interest defined in 422 by X-Window_Center, 
Y_Window_Center, Xjvidth, and Y_Height. The resulting value of the variance is 
returned to the calling program as F(N) in step 424. The sobel processing and the 
variance calculation is performed with a collection of image processing software tools 
within MVTools. MVTools is available from Coreco Imaging, US Office in 
Bedford, MA. 

Fourth Preferred Embodiment 
In the first preferred embodiment, it was disclosed how an operator could manually 
score each drop of liquid as either "CRYSTAL" or "NO CRYSTAL". In the fourth 
preferred embodiment, the operator is given a greater variety of options in deciding on 
how to score each drop. Table 1 shows listing of the operator's scoring options, 
including number, text description, and the corresponding color code. Once a micro- 
well drop has been scored a 9, the operator can further classify the crystals in a 
scoring shown in Table 2. 
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TABLE 1: 



SCORE 


DESCRIPTION 


DISPLAY COLOR 


0 


clear 


White 


1 


light precipitation 


Red 


2 


heavy precipitation 


Yellow 


3 


ugly precipitation 


Blue 


4 


phase separation 


Orange 


5 


unknown 


Violet 


6 


Spherolites 


Black 


7 


Grainy precipitation 


, Gray 


8 


Microcrystals 


Brown 


9 


Crystal 


Green 



TABLE 2: 



SCORE 


DESCRIPTION 


9.0 


crystal (no comments) 


9.1 


needles, intergrown 


9.2 


needles, single 


9.3 


plates, intergrown 


9.4 


plates, single 


9.5 


chunks, < SOmicrons, intergrown 


9.6 


chunks, < 50 microns, single 


9.7 


chunks, > 50 microns, intergrown 


9.8 


chunks, > 50 microns, single 


9.9 


gorgeous > 50 microns 



Fifth Preferred Embodiment 
In the fourth preferred embodiment, it was disclosed how an operator can manually 
score each drop of liquid into one of 10 categories with corresponding color coding, 
and how the operator can score category 9 into further subcategories of 9.0 through 
9.9. In the fifth preferred embodiment, the inspection device automatically scores 
and classifies each drop specimen by executing computer software subroutines as 
shown in FIGS. 33, 34a, 34b, 34c, 34d, and 35a and 35b under control of the program 
flow shown in FIG 36. The automatic classification can occur at three levels of detail, 
the first level, Type_of_Classification = 1, simply discriminates between a drop that is 
clear or not-clear (unknown), the second level, Type_of_Classification = 2, scores and 
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classifies the drop into classes 0 through 9 as described in Table 1 above, and the third 
level, Type_of_Classification = 3, performs second level scoring and classification, 
plus adds an additional 10 subcategories to the CLASS 9, crystal classification, as 
detailed in Table 2 above. 



Automatic Scoring and Classification 
FIG. 36 illustrates the main program flow 840 starting at step 842. The software is 
initialized with parameters, inspection lists, and Type_of_Classification detailed in 
step 486. The flow continues onto step 844 where the system moves the micro-well 
of interest within the selected micro-well plate under the selected camera. In step 
851, if the drop needs to be located, the flow continues onto step 849 wherein an 
image is acquired by the camera and software operates on the image and determines 
the location of the drop. Then a test is made to determine if the last micro-well in the 
plate has been imaged, if not then the flow loops to step 844 and continues. If the last 
micro-well in plate step 853 has been imaged then the flow continues to step 856 
where the system moves to the droplet within micro-well in micro-well plate under 
the high-resolution camera. Also in step 851 if the drop had been previously located, 
then the flow would continue from step 844 directly to step 856 without the need to 
re-locate the drop. From step 856, the flow continues onto step 857 wherein a high- 
resolution image of the drop is obtained. Then the flow goes onto step 850. In step 
850, a CALL is made to subroutine that automatically scores and classifies the drop 
depending on the Type_of_Classification required. The subroutine is detailed in FIG. 
33 and starts at step 440 in FIG. 33. After the drop has been classified the subroutine 
returns to step 852 wherein the results are stored and reported. The program flow 
continues to step 858 where a test is made to determine if the last drop in the selected 
plate has been processed, if so then the flow goes onto step 848 wherein a test is made 
to determine if the last plate has been processed. If not, the flow loops back to 856 
and continues. If the last plate has been processed, the flow goes onto step 854 and 
the program is done. 

FIG. 33 shows Micro-well Specimen Auto Score and Classify Subroutine 438 starting 
at step 440. Pointer _to_Best_Image 442 provides information to the initialization step 
444 that allows access to the image that was found to be the best focus. Plus, the 
Type_ofXlassification is passed into the routine. Alternatively, pointer 442 can point 
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to an image that was taken at a z height value known to be the focus of the system. 
After initialization 444 the subroutine 438 calculates in step 448 an 
average_gray_Background value normalized to allow for the variation present from 
various plate types, by using a first rectangular window defined by parameters shown 
in step 446 (XBackground _Center, YJBackground ^Center, X__Background 
^Width, and YJiackground Height) and by summing all of the gray scale values of 
the pixels defined by the window 446 and dividing by the number of pixels within 
that window. The average _gray_background is normalized for well-type differences 
by multiplying the calculated value by a Micro-well normer value also found in step 
446 and generally determined by measuring the various micro-well plate types under 
inspection and normalizing to the 96-well standard micro-well plate. This 
average_gray_background 448 is calculated in a window area of the image that is 
outside the area of the drop but generally within the well or within the bounding well 
walls. 

In step 450 an average ^ay_Classify_Window value is calculated in a similar 
manner as described above (except it is not normalized for micro-well type) using a 
second rectangular window defined by parameters shown in step 452 (i.e., X_Classify 
^Center, Y_Classify ^Center, X_Classify _Width, and Y_Classify Height). This 
average__gray_classify_window value 450 is taken in a rectangular window area of the 
image that is inside the area of the drop and defined by being a fraction between 0.98 
and 0.5 (with 0.8 preferred) of the width and height of the external bounding 
rectangular box from the blob utilizing subroutine mvtJ>lobjBnd. The subroutine 
mvt_blobJInd defines the extent of the drop as previously discussed in the section 
"Determining the Position of the Drop of Liquid within Each Well". 

In step 454, a diagonal difference image is calculated by stepwise subtracting pixel 
values from two pixel locations, defined by (x,y) and (x+Diff_Sep, y+DiffJSep) 
within the Classify_window. The pixel values are separated in width and height by 
the value Diff_sep from step 452. This is repeated over all pixels within the Classify 
window defined by step 452 using X_Classify ^Center, Y_Classify ^Center, 
X_Classify _Width, and Y_Classify Height. For each value calculated the absolute 
value is taken of the subtraction result and compared to a threshold value FlagJThresh 
defined in step 452. If the calculated value is greather than Flag_Thresh 452 then the 
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pixel is set at the first location in x,y equal to a value defined by Setl in step 452, if 
the calculated value is equal to or less than FlagLThresh, 452, then the pixel value is 
set to zero. This can be seen by the mathematical equations and flow described in 
step 454 in calculating a Diffjmage_l. Typical values for DiifjSep are between 1 to 
20 pixels with 7 preferred. Typical values for Setl are between 1 and 511 with 128 
preferred. Typical values for Flag^Thresh are between 5 and 50 with 25 preferred. 

In Step 456, a calculation similar to that performed in step 454 is performed on the 
Classify_Window except that the separation between the two pixels undergoing the 
calculation is defined by (x+Diff_Sep, y) and (x, y+DiffJSep), as is shown in the 
mathematical calculation in 456 to generate a DifFJmage_2. This calculation uses 
definitions shown in step 452. Typical values for Set2 are between 1 and 511 with 
200 preferred. 

In Step 458 the Classifyjbnage, which is a combination of the images generated in 
step 454 and 456 is calculated as shown by the mathematical equations shown in step 
458 using definitions shown in step 452. If the x,y pixel value in either Di£f_Image_l 
or Diff_Image_2 (steps 454 and 456 respectively) has a value equal to Setl 452 then 
the pixel value is set at (x,y) in Classifyjmage equal to Set2 452. Otherwise, the 
value is equal to zero(0) as shown in the mathematical equations in step 458. The 
calculations are repeated for all pixels within the window defined in 452. The 
Classifyjmage is basically an image of the classify_image_window wherein edges 
present within the original Bestjtaaage are detected. 

In step 460, the value of number_Pixels_Set2 is set equal to the total number of pixels 
that are set equal to Set2 452 in step 458. Also, the value of Total_Pixels_in_Window 
is set to the total number of pixels in the Classify_window in step 458 . 

In step 462, a Score J3ray is calculated by dividing the 
Average_Gray_Classify_Window determined in step 450 by the 
AverageGrayJBackground found in step 448. A Score JFlag is also calculated by 
dividing the Number_PixelsJ3et2 by TotalJPixels_in_Window from step 460. The 
Score_Gray and Score_Flag are normalized in this matter. 
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In step 464, the values of Score J3ray, ScoreJFlag, and Type_of_Classification are 
passed to a classify subroutine and a classification is returned for the Classifyjmage, 
effectively classifying the protein crystals within the window. Details of the Classify 
subroutine 464 are provided in FIG 34a, 34b, 34c,and 34d, plus FIG 35a and 35b. 

FIG 34a shows the Classify Subroutine 464, Start of Classify subroutine is shown in 
step 468 followed by initialization 470 whereby the initial classification CLASS value 
is set to 5, representing "unknown" and the flow goes onto step 480. 

Step 480 calculates whether the drop is clear and the CLASS = 0 by a test detailed in 
step 480 using thresholds defined in step 482 (Clear JFlagL.LT and Clear_Gray_GT) 
with the following equation: if score_flag is less than Clear JFlagJLT and Score J3ray 
is greater than Clear J3ray__GT then set CLASS = 0. At step 481, a test is made to see 
of the type_of_classification is equal to 1, the first classification type wherein the drop 
* is classified as simply clear (0) or unknown (5) as previously discussed. If the 
Atype_of_classification is equal to 1 then the flow goes on to step 483 and then returns 
to FIG. 33 step 466 with the results. If the type_of_classification is not equal to 1 
then the flow goes onto step 476 for further classification. 

Step 476, utilizing threshold value parameters shown in step 478 
(LgtJPrecip JFlagJLT, Lgt^Precip^Flag^GT, Lgt_Precip_Gray_GT), assigns the 
value of 1 to CLASS indicating that Light Precipitation is present in the 
dassifyjfaiage. Step 476 utilizes the mathematical equation which states if 
Score_Flag is less than Lgt_Precip_Flag_LT and Score_Flag is greater than 
Lgt_PrecipJFIag_GT and Score_Gray is greater than LgtJPrecip_Gray_GT then set 
CLASS to value 1. 

Step 484 calculates heavy precipation by using thresholds detailed in step 486 
(Heavy_Precip_FlagL_LT, Heavy_Precip_Flag_GT, Heavy_Precip„Gray_LT) with 
the following equation: if score_flag is less than Heavy_Precip_FlagJLT and 
scorejlag is greater than Heavy J>recipJFlagj3T and score_gray is less than 
Heavy J>recipj3rayJLT then set CLASS = 2. 
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Step 488 calculates ugly precipation by using thresholds detailed in step 490, 
UglyJPrecipJFlagJLT, UgIy_Precip_Flag^GT, UglyJftecip_GrayJLT, with the 
following equation: if score Jlag is less than UglyJPrecip _Flag_LT and scorejlag is 
greater than UglyJPrecipJFlagJjT and score jjray is less than 
Ugly_Precip_Gray_LT then set CLASS = 3. 

Step 492 continues the classification process in FIG 34b. 

In FIG 34b, the continuation of the classification process 700 is shown continuing in 
step 702. Step 704 calculates micro_crystals by using thresholds detailed in step 703 
(Micro_CryJFlag_GT and Micro_Cry_Gray_LT) with the following equation: if 
score jQag is greater than Micro_Cry_Flag_GT and score _gray is less than 
Mtcro_Cry_Gray_LT, then set CLASS = 8. 

Step 704 calculates crystals by using thresholds detailed in step 705 
(Crystal_Flag_.LT, Crystal JFlag^GT, and Crystal JjrayJjT) with the following 
equation: if scorejQag is less than Crystal_Flag_LT and score_flag is greater than 
Crystal JFlag^GT and score _gray is greater than CrystalJ3ray_GT then set CLASS = 
9. 

Step 706 calculates Grainy precipitation by using thresholds detailed in step 707 
(GrainyJFlag^GT, and Gramy_Gray_GT) with the following equation: if CLASS = 
8 and score_Jlag is greater than Grainy_Flag^GT and score _gray is greater than 
GrainyJ3ray_GT, then set CLASS = 7. 

Step 708 continues the classification process onto FIG 34c as 725. 
Step 726 continues from step 708 of FIG 34b. 

In FIG 34c, Step 727 calculates and generates a first set of additional image features 
for further use in classification taking as input 726 (X_Classify ^Center, Y_Classify 
^Center, X_Classify _Width, Y_Classify Height, Pointer Jo_classifyjmage, and 
mvtJ>lob_analysisJ?arams). In step 727, M\T_Tools_Blob_Analysis with pointer to 
window in Classifyjmage is called. Step 727 gets the num_found of blobs and, for 
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each blob found, step 727 gets its area(m), height(m), width(m), perimeter(m), and 
location of each as location_X(m) ands location_Y(m). These values are recorded. 
These calculations are performed on the image called Classifyjmage, which is the 
image that was formed in FIG 33 as step 458. 

In step 728, if numjfound is not greater than zero(0) the subroutine goes to step 738. 
However, if any blobs are found then further analysis is started in step 729 by setting 
m=l, num_spherolite = 0, and spherolite(m) = 0. In step 730, the following 
blobjratios are calculated: Circle_Like_HW(m) = height(m)/Width(m) and 
Circle JLikejVHW(m) = area(m)/(height(m)*width(m». For blobs that are circular, 
CkcleJLikeJHW will be around a value of one (1). If blobs get elongated then the 
value will be other than one. CircleJLike_AHW for circular blobs has a value 
around 0.785. For square-like blobs the value will be closer to one (1). The program 
flow goes onto step 732. 

Step 732 determines whether to classify a drop as having a spherolite by utilizing the 
parameters found in 734 (Circle_Like_HWJower, CncleJJkeJHWJJpper, 
Circle_Like_AHW_Lower, and Circle JLike_AHWJJpper). The following equation 
is used:: if CLASS « 8 or 9 and Circle_Like_HW(m) is less than 
Circle_Like_HW_Upper and Circle_Like_HW(m) is greater than 
CircleJLikeJHTWJower, and Circle JJke_AHW(m) is less than 
Circle_Like_AHWJJpper and Circle_Like_AHW(m) is greater then 
Circle_Like_AHWJLower, then num_spherolite = num sperolite +L An increment 
is done by calculating m = m+1 and setting spherolite(m) = 1 to show one has been 
found at m. 

Step 736 tests whether all of the found blobs have been classified by testing m against 
num_found and, if equal, the subroutine goes onto step 738. If not the program loops 
back to step 730 and flows through as above. 

Step 738 goes onto FIG 34d in the classification subroutine as 740. 

FIG 34d continues from FIG 34c at 742 and goes to 744 where a second set of 
additional features is calculated. These features are generated with the original 
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Bestjmage generated in step 328 (FIG. 31) as the image of the best focus. For these 
additional features, the results are used from the blob analysis performed in FIG. 34c 
on the classifyjmage. Such features as shown in parameters 746 as numjbund, 
area(m), height(m), width(m), perimeter(m), location_X(m), locationY(m), 
num_spherolite, and spherolite(m). Plus it uses the average_gray background from 
FIG. 33 step 448, and the pointer JoJBestJSmage in FIG. 31 step 328. In step 744 
"m" and numj>hase_sep are set equal to zero. 

In step 748, if the number of spherolites previously found (num_spherolite) is not 
greater than zero (0), then the flow goes to step 756 and the remaining steps shown in 
FIG 34d are bypassed. But if the num_spherolite is greater than zero (0), then the 
flow goes onto step 749 wherein m is incremented by 1. Then the flow goes on to 
step 751 to test whether to go to step 756 or to step 753 depending on comparing "m" 
to the num_found. In step 753, the value of spherolite(m) found in FIG. 34c step 732 
is tested to see if any were classified as spherolite. If not, then the flow loops back to 
step 749, If spherolite(m) is equal to one (1) then step 750 is executed. 

In step 750, the information location_X(m), location_Y(m), one half of height(m) 
and one half of width(m) is used to calculate the average gray value within this 
reduced image inside each spherolite. The Best_hnage pixel data obtained from the 
previous blob analysis is utilized and phasejsep_Gray(m) is set equal to this value. 
Phase jsep_Gray(m) is then normalized by dividing it by the 
Average JjrayJBackground. The program flow goes onto step 752, wherein 
phase jsep_Gray(m) is tested to see if it is greater than a parameter 
phase_sepJ3ray_GT from parameter input step 754. If true, then CLASS is set equal 
to four(4) and the program loops up to step 749. 

In step 751 the automatic program has classified the microdrop into one of the 10 
primary classes, 0 through 9, detailed previously in Table 1. Then, a test is made to 
see if the type ^^classification is equal to 2. If so, then the flow goes onto step 755 
wherein the flow is returned to FIG. 33. In step 466, a general classification is 
complete. If further classification into subcategories is required, then the flow goes 
onto step 758. 
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In FIG. 35a, a crystal classification subroutine 762 further classifies any CLASS 9 
crystal image into to additional subclasses, 9.2, 9.4, 9.6, 9.8, or 9.9, as previously 
discussed in Table 2. The subroutine begins at step 760 and goes onto step 764. In 
step 764, the blob counter m is set to zero. A test is conducted to see if the CLASS is 
equal to 9 and, if not, the flow goes to step 778 and returns to FIG. 33 step 466. Step 
764 uses input values as shown in step 766 (Average_Gray_Background, 
Pointer_To_Best_Image, num_found, area(m), height(m), width(m), perimeter(m), 
location_X(m), location_Y(m), num_spherolite, and spherolite(m)), as previously 
described. If the CLASS is equal to 9 another test is made to see if one or more than 
one blobs were previously found. If only one blob was previously found then the 
flow goes onto step 768. If more then one is found, the flow goes onto step 769 in 
FIG 35b. In step 768, a height to width ratio is calculated and compared to thresholds 
representing needle-like characteristics in step 770. If the test conditions are met, 
then the CLASS is set to 9.2. If not, the flow goes onto step 771. In step 771, the 
height to width ratio is compared to thresholds representing plate-like characteristics 
in step 772. If the conditions of the test are met, the CLASS is changed to 9.4 and the 
flow goes onto step 773. If not, no further classification is performed and the flow 
goes onto step 778 and the subroutine returns to FIG. 33 step 466. In step 773, a 
normalized average gray value, Chunk_Gray(m), within the blob is calculated and the 
flow goes onto step 774. In step 774, if Chunk^yCm) is less than a threshold 
Chunck_Gray_LT (from step 775), then CLASS is set to 9.6 and the flow goes onto 
step 776. In step 776, if Chunk _gray(m) is greater than a threshold Chunk_50_GT, 
(from step 777), then CLASS is set to 9.8 and the flow goes onto step 780. In step 
780 if area(m) is greater than threshold gorgeous_GT (from step 779), then CLASS = 
9.9 and the flow goes onto step 778. Then the subroutine returns to FIG. 33 step 466. 

FIG 35b shows flowchart 781 illustrating the further classification of a CLASS 9 
image having multiple blobs within the image into additional subclasses, 9.1, 9.3, 9.5, 
or 9.7, as previously discussed in Table 2. The subroutine begins at step 786 from 
step 769 in FIG. 35a and goes onto step 782. In step 782, a loop begins at m equals 0 
for each blob m using input values from step 784 (Average J3rayJ3ackground, 
Pointer_To_Best_Image, num_found, area(m), heighftm), width(m), perimeter(m), 
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location_X(m), location_Y(m), num_spherolite, and spherolite(m)) as previously 
described The flow goes onto step 788 where m is incremented and a height to width 
ratio is calculated and compared to thresholds representing needle-like characteristics 
in step 790. If the test conditions are met, then the CLASS is set to 9.1. If not, the 
flow goes onto step 794. In step 794, the height to width ratio is compared to 
thresholds representing plate-like characteristics in step 796. If the conditions of the 
test are met, the CLASS is changed to 9.3. If not, no further classification is 
performed and the flow goes onto step 812 to determine if all of the blobs have been 
tested. In step 798, a normalized average gray value (Chunk_Gray(m) as described 
before) within the blob is calculated and the flow goes onto step 802. la step 802, if 
Chunk_gray(m) is less than a threshold Chunk_GrayJLT (from step 804) then 
CLASS is set to 9.5 and the flow goes onto step 806. In step 806, if Chunk __gray(m) 
is greater than a threshold Chunck_50j3T (from step 808) then CLASS is set to 9.7 
and the flow goes onto step 812. In step 812, if all of the blobs have been tested, the 
flow goes onto step 810 and the subroutine returns to FIG. 33 step 466, If not all 
tested, then the flow loops back to step 788 for the next blob and the process loops 
until complete. 

Typical parameter and threshold values for use in classification in FIGS. 33, 34a, 34b, 
34c, 34d, and 35a and 35b are given in Table 3 along with the preferred value. These 
values serve only as a guide, and other values may be used when circumstances 
justify. For example, different lighting conditions, variations in the transparency of 
micro-well plates, variations in the formulations of the protein growing media and 
drop, values may be used as well. One skilled in the art may adjust the parameter and 
threshold values to tune in the classification results specific to their setup. 



Table 3 Typical and preferred values for threshold and Classification parameters 
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Prototype 

Applicants have designed, built and tested a working prototype of the present 
invention. 

FIG. 30 shows the major components of the Applicant's prototype. Proteomic crystal 
verification and inspection system 100 has three axis of linear motion. Linear actuator 
115 is preferably linear actuator model # 802-0763D available from Dynamic 
Solutions of Irvine CA., with 600 mm of travel driven by an enclosed 0.5 mm pitch 
ballscrew. Linear actuator 115 is driven by an intelligent self-contained servo motor 
110 model # SM2320 SQ available -from Animatics Corp. of Santa Clara, CA., with 
38 oz-in of available torque. Servo motor 110 communicates with a Windows Based 
computer 105 through a serial connection routed through a central control unit 190. 

Linear actuator 115 has stationary part 116 fixed to a granite table top 170. Motor 
1 10 moves moving part 117 along the x-axis. Granite top 170 is supported by a frame 
180. Frame 180 has casters and adjustable legs. Plate 127 is attached to moving part 
117. At each end of plate 127, a spacing block 128 spaces fixture plate 129 from 
plate 127. At each of its ends, fixture plate 129 is supported by spacing block 128. 
Fixture plate 129 provides for the mounting, removal, and positioning of multiple 
micro-well plates 125A - 125F. Preferably micro-well plates 125A - 125F are agar 
plates, micro-titer well plates of 96, 384, 1536 wells, or other sample plates. 

Support 191 is positioned adjacent to fixture plate 129 and contains a light source. In 
the preferred embodiment, the light source is model # A08925 fiber-optic back light 
available from Aegis Electronics of Carlsbad, CA. 

In the preferred embodiment, linear actuator 150 is model # 802-1384A available 
from Dynamic Solutions of Irvine CA., with 350 mm of travel driven by an enclosed 
0.5 mm pitch ballscrew. Linear actuator 150 has stationary part 153 horizontally 
bridged above linear actuator 115 and supported by pillars 152. Moving base 154 
provides a mounting base for components of linear actuator 160. In a preferred 
embodiment, linear actuator 150 is driven by driven by an intelligent self-contained 
servo motor 120 identical to motor 110. Servo motor 120 communicates with 
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Windows Based computer 105 through a serial connection routed through central 
control unit 190. 

In a preferred embodiment, linear actuator 160 is model # 802-0756D available from 
Dynamic Solutions of Irvine CA., with 100 mm of travel driven by an enclosed 0,5 
mm pitch ballscrew. Linear actuator 160 has with a stationary part 161 mounted 
perpendicular to and fixed to moving base 154 of linear actuator 150. Linear actuator 
160 is driven by intelligent self-contained servo motor 130. Servo motor 130 is 
preferably identical to motor 110. Servo motor 130 communicates with the Windows 
Based computer 105 through a serial connection routed through the central control 
unit 190. 

Moving plate 162 provides a mounting base for camera 155. In a preferred 
embodiment camera 155 is a high-resolution monochrome megapixel CCD camera 
' model # CVM1 from JAI America INC. of Laguna Hills, CA. Camera 155 has lens 
165. Preferably, lens 165 has a 0.75x to 3x zoom capability and is model # NT52-571 
available from Edmund Industrial Optical of Barrington, NJ. Moving plate 162 also 
-provides a mounting base for camera 135. Preferably, camera 135 is a high-resolution 
monochrome megapixel CCD camera model # CVM1 from JAI America INC. of 
Laguna Hills, CA. Camera 135 has lens 145. Preferably, lens 145 has a 2.5x to lOx 
zoom capability and is model # NT54-396 available from Edmund Industrial Optical 
of Barrington, NJ. 

Moving plate 162 also provides a mounting base for a zoom lens motor 192, an 
intelligent self-contained servo motor model # SM2315D available from Animatics 
Corp. of Santa Clara, CA., with 20 oz-in of available torque. The servo motor 
communicates with the Windows Based computer 105 through a serial connection 
routed through the motion control box 190. The zoom lens motor operates the zoom 
lens 145 through a conventional belt drive. 

A bar-code reader 175 is mounted adjacent linear actuator 115 and is attached to 
support 172 fixed on granite top 170. Bar-code reader 175 is positioned to view a 
bar-code identity label attached to a micro-well plate when the micro-well plate is 



29 



WO 03/064047 PCT7US03/02107 

positioned under linear actuator 150. Preferably, bar-code reader 175 is model # 
BL601 available from Keyence Corporation of America of Newark, NJ. 



A plate sensor transmitter/receiver 186 is also fixed to support 172 and is aligned to 
sense whenever a micro-well plate 125 breaks a beam of light emitted by 
transmitter/receiver 186 and reflected by a reflector 188. Reflector 188 is mounted on 
support 191 on the opposite side of the linear actuator 115. Plate sensor 
transmitter/receiver 1 86 and reflector 188 are preferably model # E3T-SR13 available 
from Western Switch Controls of Santa Ana, CA 

Block Diagram Showing Connectivity of the Prototype 
FIG. 6 shows a block diagram illustrating the connectivity of Applicant's prototype. 
Linear actuator motors 110, 120 and 130, and zoom motor 192 receive DC power 
from 48 Volt DC power supply 680 through an electrical connection. Linear actuator 
motors 110, 120 and 130, and zoom motor 192 motors communicate with Windows- 
based computer 105 through common serial line 633. Bar-code reader 175 
communicates with computer 105 through a communications line and plate sensor 
186 communicates with the computer 105 through a communications line. Monitor 
620 displays information to the operator transmitted from computer 105. Cameras 
135 and 155 communicate with frame grabber 630 through communication lines. 
Frame grabber 630 is installed within computer 105 and is preferably PCVision from 
Coreco Imaging US Office in Bedford, MA.. Frame grabber 630 digitizes the image 
from the camera to form the digitized image data array within computer 105. A 4- 
port Ethernet hub 640 provides for connectivity between computer 105, central 
control unit 190, and an external Ethernet 670. By providing for connectivity to the 
Ethernet, computer network communications are possible. The central control unit 
190 controls light source 194 through an analog control line. Central control unit 190 
receives 24 volt DC power from 24 volt DC power supply 660. Emergency stop 
button and switch (e-stop) 650 is connected to central control unit 190. 

Experimental Results 

Fifty-three test images were obtained from the system and were both automatically 
classified by the system and were manually classified by four scientists. Table 4 
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shows the correlation percentage between the various scientists and the automatic 
classification provided by the system. 



Table'4 
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Utilization of Color 

In another preferred embodiment, the present invention is configured to record color 
images. It is desirable to be able to analyze color images in that certain precipitation 
products in the protein crystallization process have distinctive colors and a 
, crystallographer or automated image analysis algorithm may use the color information 
to help discriminate crystallization results. 

True Color Picture 

FIG. 37 shows a side view of micro-well plate 125A positioned on fixture plate 129. 
Support 191 with embedded light source 194 is positioned to the side of fixture plate 
129. Light from light guide 195 is directed upward through cutout 131. Light guide 
195 is positioned between fixture plate 129 and plate 127 such that both plates can 
move around the light guide 195 without interference. 

Linear polarized filter 352 is rotationally mounted above light guide 195 such that 
light from light guide 195 can be polarized linearly at a programmable angle before it 
transits through micro-well plate 125A. Polarizer drive belt 356 (top view shown in 
FIG. 38) rotates polarized filter 352 about a vertical axis. Polarization drive belt 356 
is driven by motor 358. Motor 358 is controlled by CCU 190 (FIG. 40). Second 
filter 354 (top view shown in FIG. 39) is positioned above micro-well plate 125A 
such that light transiting through micro-well plate 125A goes through second filter 
354 before it goes into the camera zoom lens 145. Second filters 354 are mounted on 
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filter wheel apparatus 355. Filter wheel apparatus 355 rotates the operator selected 
second filter 354 into position under the zoom lens 145. The selected second filter 354 
is preferably either a red, green or blue dichroic filter. Preferably, individual images 
taken through the red, green and blue second filters are combined to form a true color 
image. 



False Color Image 

In addition to the true color images that may be formed using red, green, and blue 
filters 354, a false color (also called a pseudo color) image may be formed by taking 
three individual images using linear polarized filter 352 at three different polarization 
angles with respect to a second filter, hi this preferred embodiment, a linear polarized 
filter is substituted for the dichroic second filters 354 discussed above. For example, 
the polarized axis may be at 90 degrees to each other, and at plus and minus 45 
degrees to each other. The three images are then called red, green, and blue and a 
false color image is produced. If the crystal exhibits any polarization rotation effects, 
then a very colorful image results. This pseudo color image is useful in detecting very 
small and fine crystals from the image background material. Other polarizing angles 
may be selected as well. 

In the preferred embodiment, ligjit guide 195 is model # A08925 fiber-optic backlight 
available from Aegis Electronics of Carlsbad. CA. Light source 194 is a Dolan- 
Jenner Model-PL-900 available from Edmund Industrial Optics, Bairington, NJ. First 
polarized filter 352, second filter 354 (including the linear polarized filters, and the 
dichroic red, green, blue filters) are available from Edmund Industrial Optics, 
Barrington, NJ. Filter wheel 355 is model FW1 available from Integrated Scientific 
Imaging Systems, Inc of Santa Barbara, CA. 

Combined with an Automated Tray Storage and Retrieval System 
In another preferred embodiment, proteomic crystal verification and inspection 
system 100 (FIG. 30) is operated in conjunction with automated storage and retrieval 
system 900, as shown in FIGS. 41 A and 41B. FIG. 41A shows a front view and FIG. 
41B shows a back view of the prototype automated storage and retrieval system 900 
that Applicant' s have built and tested. Also show are two proteomic crystal 
verification and inspection systems 100. In the preferred embodiment, an operator 
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loads micro-well plate holding tray 901a (FIG. 42) into access drawer 902 (FIG. 
41A). Preferably, micro-well plate holding tray 901a (FIG. 42) holds 6 micro-well 
plates that contain solution in which protein crystals are growing. Storage gantry 903 
(FIG. 41B) removes tray 901a from access drawer 902. As shown in FIGS. 41A and 
41B, automated storage and retrieval system 900 has a large number of storage slots. 
After removing tray 901a, storage gantry 903 transfers tray 901a to one of the storage 
slots. Preferably, this process is repeated until a plurality of micro-well plate holding 
trays are stored in a plurality of storage slots. Then, at defined time intervals, storage 
gantry 903 sequentially transfers micro-well plate holding trays from the storage slots 
to work cell area 904. An enlarged perspective view showing work cell area 904 is 
shown in FIG. 43. Once a micro-well plate holding tray has been placed in work cell 
area 904, work cell gantry 905 will sequentially remove micro-well plates from the 
tray and place them onto either one of the two proteomic crystal verification and 
inspection systems 100. Each well in the micro-well plates will then be inspected in a 
^fashion similar to that described in the above preferred embodiments. After the 
micro-well plates have been inspected, work cell gantry 905 will place the micro-well 
plates back into their micro-well plate holding trays. Then, storage gantry 903 will 
return the micro-well plate holding trays back to their storage slots. 

Sequence of Operation 
Storing the Mircowell-Plates 
FIG. 45 shows a front view similar to the perspective view shown in FIG. 41A. 
Access drawer 902 is closed. 

FIG. 46 shows a top view of access drawer 902. In FIG. 46, an operator has opened 
access drawer 902. ' 

In FIG. 47, the operator has placed trays 901a and 901b onto access drawer 902. 
Trays 901a and 901b each contain six 96-well micro-well plates. Inside each well of 
each micro-well plate is a solution in which the operator is attempting to grow protein 
crystals. 

In FIG. 48, the operator has closed access drawer 902. Sensors 922a and 922b are 
situated adjacent to drawer 902 are directed to sense the presence of a tray on access 
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drawer 902. In the preferred embodiment, sensors 922a and 922b are MC-S series 
safety interlock switch, control unit and sensors manufactured by Scientific 
Technologies Inc. with offices in Freemont, CA Upon sensing the presence of trays 
901a and 901b, sensors 922a and 922b send signals to CPU 921 (FIG. 44). CPU 921 
is a windows based computer that has been programmed to operate automated storage 
and retrieval system 900. 

FIG. 49 shows a back view of automated storage and retrieval system 900 similar to 
the perspective view shown in FIG. 41B. In FIG. 49, trays 901a and 901b are resting 
on access drawer. 

In FIG, 50, CPU 921 (after having received the signals from sensors 922a and 922b) 
has sent a signal to linear actuator motor 923a (FIG. 44). linear actuator motor 923a 
is connected to linear actuator 923b directly and to linear actuator 923c through 
actuator drive shaft 923d. In the preferred embodiment, linear actuators 923b and 
923c are belt driven linear actuators, Series HUB manufactured by Parker 
Automation- Daedal Division with offices in Lrwin, PA. In FIG. 50, linear actuators 
923b and 923c have moved storage gantry 903 to the left above tray 901a. 

In FIG. 51, CPU 921 has sent a signal to linear actuator motor 924a. Linear actuator 
motor 924a is connected directly to belt driven linear actuator 924b. In the preferred 
embodiment, linear actuator 924b is Series HLE manufactured by Parker Automation- 
Daedal Division with offices in Irwin, PA Linear actuator 924b controls the vertical 
motion of platform 925. Platform 925 is also supported by support bearing 924c. In 
F?G. 51, linear actuator 924b has lowered platform 925 so that is ^proximately 
adjacent to tray 901a. Note that as platform 925 has lowered, counterweight cylinder 
926 has risen. Counterweight 926 rides on bearing 927 and is connected to platform 
925 by cable 928. Cable 928 runs through pulleys 929a and 929b. In the preferred 
embodiment platform 925 weights approximately 70 lbs. (fully loaded) and 
counterweight 926 weights approximately 50 lbs. By connecting counterweight 926 
to platform 925, linear actuator 924b has to do less work to move platform 925. 
Therefore, the vertical movement of platform 925 is smoother. 
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FIG. 52 shows a top view of the position of platform 925 in FIG. 51. Linear actuator 
motor 930a is connected directly to linear actuator 930b. Linear actuator motor 930a 
and linear actuator 930b are both supported by platform 925. In the preferred 
embodiment, linear actuator 930b is a gear driven linear actuator, ER Ball-screw 
series manufactured by Parker Automation- Daedal Division with offices in Irwin, 
PA. Linear actuator 930b controls the horizontal movement of robotic gripper motor 
931a and robotic gripper 931b. Robotic gripper motor 931a controls robotic gripper 
931b. 

In FIG. 53, CPU 921 (FIG. 44) has sent a signal to linear actuator motor 930a, causing 
linear actuator 930b to move robotic gripper 931b closer to tray 901a. Robotic 
gripper has gripped tray 901 a. 

In FIG. 54, robotic gripper 931b has pulled tray 901a along tracks 933 onto platform 
925. Bar code reader 934 is attached to the side of platform 925 and is orientated so 
that it is able to read the bar code of each micro-well plate as it passes by. The bar 
code information is transmitted to CPU 921 where it is stored in the computer's 
database (FIG. 44). 

In FIG. 55, storage gantry 903 has moved tray 901a to its storage slot in storage rack 
932. While at the location shown in FIG. 55, linear actuator motor 930a moves 
robotic gripper 931b (FIG. 52) towards storage rack 932 so that tray 901a is stored 
inside its storage slot. 

hi FIG. 56, after having deposited tray 901 a in its storage slot, storage gantry 903 has 
moved adjacent to tray 901b. 

In FIG. 57, storage gantry 903 has moved trays 901a - 901aab to their assigned 
storage slots in storage rack 932 in a fashion similar to that describe above in 
reference to FIGS. 44 - 56. Although FIG. 57 shows just fifty-four trays actually in 
storage rack 932, in the preferred embodiment, storage rack 932 can hold a total of 
1692 trays. Also, in the preferred embodiment, each tray can hold six micro-well 
plates. Therefore, in the preferred embodiment, storage rack 932 can hold a total of 
10,152 micro-well plates. Moreover, as previously noted in reference to FIG. 54, 
CPU 921 records the bar code of each micro-well plate as its being loaded onto 
platform 925. Also, in the preferred embodiment, CPU 921 records the location of 
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the storage slot in which each micro-well plate has been located. Therefore, for 
example, an operator can immediately ascertain the location of a particular micro-well 
plate by referring to the database of CPU 921 . 

Moving the Micro-well Plates to the Work Cell Area 
la the preferred embodiment, by inserting commands into CPU 921 via keyboard 934 
(FIG. 44) an operator can inspect any given micro-well plate at any moment. Or, 
CPU 921 can be programmed to automatically cause automated storage and retrieval 
system 900 (FIG. 58) to transfer a designated micro-well plate to proteomic crystal 
verification and inspection systems 100 at a specific designated time or at designated 
time intervals. For example, in a preferred embodiment, CPU 921 has been 
programmed to cause micro-well plates stored on trays 901a - 901d to be inspected 
every two weeks for protein crystal growth without any operator intervention. 

In FIG. 58, storage gantry 903 has moved to the top of tray stack 935 containing trays 
901a - 901aab. While at this location, micro-well plate 901a will be loaded onto 
platform 925 in a fashion similar to that described above. 

In FIG. 59, storage gantry 903 has moved platform 925 containing micro-well plate 
901a in front of work cell area 904. 

FIG. 60 shows a top view of platform 925 in front of work cell area 904. Work cell 
area 904 contains platform 936. At the end of platform 936 are robotic grippers 937b 
having robotic gripper motors 937a. Robotic gripper motors 937a are controlled by 
CPU 921 (FIG. 44). Adjacent to platform 936 are fixture plates 129 of proteomic 
crystal verification and inspection systems 100. A preferred proteomic crystal 
verification and inspection system 100 is discussed above and shown in greater detail 
in FIG. 30. Work cell gantry 905 surrounds work cell area 904. 

In FIG. 61, linear actuator 930b has moved tray 901a onto platform 936. Robotic 
gripper 937b has gripped the forward end of tray 901a 

In FIG. 62, robotic gripper 924b has released the back end of tray 901a so that tray 
901a has been left in position on platform 936. 
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In FIG. 63, trays 901b, 901c and 901d have been stored on platform 936 in a fashion 
similar to that described in reference to tray 901a 

In FIG. 64, linear actuator 938b of work cell gantry 905 has repositioned linear 
actuator 939b. Linear actuator motor 938a is connected to linear actuator 938b 
directly and to linear actuator 938d through actuator drive shaft 938c. In the preferred 
embodiment, linear actuators 938b and 938d are belt driven linear actuators, model 
number Series HLE manufactured by Parker Automation- Daedal Division with 
offices in Irwin, PA 

In FIG. 65, linear actuator 939b has moved gripping device 941 over a micro-well 
plate stored on tray 901a. Linear actuator 939b is controlled by linear actuator motor 
939a In the preferred embodiment, linear actuator 939b is a gear driven linear 
ractuator, ER Ball-screw Series manufactured by Parker Automation- Daedal Division 
.with offices in Irwin, PA. 

FIG. 66 shows a side view of the situation in work cell area 904 shown in FIG. 65.- 
Gripping device 941 is positioned over a micro-well plate stored on tray 901a. 
Gripping device 941 has linear actuator 942b controlled by linear actuator motor 
942a . In the preferred embodiment, linear actuator 942b is a gear driven linear 
actuator, ET Ball-screw Series manufactured by Parker Automation- Daedal Division 
with offices in Irwin, PA. Linear actuator 942b controls the vertical movement of 
robotic gripper 943b. The gripping movement of robotic gripper 943b is controlled 
by robotic gripper motor 943a 

In FIG. 67, linear actuator 942b has lowered robotic gripper 943b towards tray 901a 
Gripper 943b has gripped the micro-well plate on tray 901a 

In FIG. 68, gripper 943b has removed the micro-well plate from tray 901a 

In FIG. 69, linear actuator 939b has moved robotic gripping device 941 so that the 
micro-well plate is positioned over fixture plate 129. 
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In FIG. 70, gripper 943b has lowered the micro-well plate onto fixture plate 129. 



In FIG. 71, gripper linear actuator 942b has raised gripper 943b, leaving behind the 
micro-well plate on fixture plate 129. 

FIG. 72 shows a top view of the situation depicted in FIG. 71. Robotic gripping 
device 941 is positioned over fixture plate 129. 

In FIG. 73, all the micro-well plates stored on tray 901a have been transferred to 
fixture plate 129. 

In FIG. 74, fixture plate 129 is moving the micro-well plates underneath lens 944 of 
proteomic crystal verification and inspection system 100. The operation of fixture 
plate 129 and proteomic crystal verification and inspection systems 100 is described 
in detail above in reference to FIGS. 1 - 40. Robotic gripping device 941 has moved 
so that it is depositing a micro-well plate stored on tray 90 Id onto the second fixture 
plate 129. 

In FIG. 75, first fixture plate 129 has continued sequentially moving micro-well plates 
under lens 944. Robotic gripping device 941 is in the process of depositing micro- 
well plates on second fixture plate 129. 

In FIG. 76, second fixture plate 129 is sequentially moving micro-well plates under 
lens 944. Robotic gripping device 941 is in the process of removing micro-well plates 
from first fixture plate 129 and placing them back onto tray 901a. 

La FIG. 77, all micro-well plates on trays 901a through 901d have been inspected by 
proteomic crystal verification and inspection system 100 in the manner described 
above. All micro-well plates are now back on trays 901a through 901d. Storage 
gantry 903 (FIG. 41B and FIG. 49) will now move trays 901a through 901d back to 
their original positions in storage rack 932 (FIG. 57) until they are scheduled to be 
inspected again in accordance with programming input into CPU 921 (FIG. 44). 

Operator Retrieval of Stored Tray 
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In the preferred embodiment, an operator has the option of personally inspecting or 
handling a micro-well plate that has been stored in storage rack storage rack 932 
(FIG. 55). The operator first inputs the micro-well plate's identifying bar code into 
CPU 921 via keyboard 934 (FIG. 44). CPU sends a signal to storage gantry 903 (FIG. 
41B, FIG. 59) and the appropriate tray containing the requested micro-well plate is 
removed from storage rack 932. Storage gantry 903 then delivers the tray containing 
the requested micro-well plate to access drawer 902 (FIG. 41 A, FIG. 46). The tray is 
loaded onto the access drawer 902. Once the tray is on access drawer 902, the 
operator can personally retrieve the tray by opening the drawer, as shown in FIG. 46. 
After he is finished with the tray, the operator returns the tray by placing it back on 
access drawer 902. Tray sensors 922a and 922b (FIG. 48) will sense the presence of 
the tray and CPU will send a signal to storage gantry 903 to grip the tray. Bar code 
reader 934 (FIG. 54, FIG. 44) will read the bar code of the micro-well plates on the 
tray and storage gantry 903 will return the tray to its appropriate storage slot in 
storage rack 932. 

' '"V 

Preferred Micro-well plate Holding Tray 
Although a variety of designs for tray 901a will work in conjunction with the present 
invention, a preferred micro-well plate holding tray 901a can be seen by reference to 
FIGS. 78 - 82. FIG. 78 shows a perspective view of a preferred micro-well plate 
holding tray 901a with a left-rail 906a, a top-rail 906b, a right-rail 906c, and a bottom- 
rail 906d. In the preferred embodiment the rectangular dimensions of tray 901a are 
approximately 58 mm by 148 mm. The rails are preferably designed to be at or below 
the height of a typical micro-well plate. In the preferred embodiment, they are 
approximately 14 mm in height. Also, in the preferred embodiment, tray 901a can 
store six micro-well plates. The storage tray is molded in one-piece of thermoplastic 
resin and can be clear or colored. Color coding of the trays adds visual feedback for 
sample identification. Six rectangular pockets 907a - 907f (each one designed to hold 
a standard micro-well plate) are formed in tray 901a, between rails 906a - 906d and 
interior rails 908a - 908d. Cut-down area 909 centered in top-rail 906b provides 
access for a robotic gripper. For example, a robotic gripper can grasp top rail 906b at 
cut-down area 909 and pull it or push it from or into one of the storage slots shown in 
automated storage and retrieval system 900 (FIGS. 41 A and 41B). Thermoplastic tray 
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901a is smooth on its bottom side and can therefore efficiently slide over supporting 
surfaces so that the samples contained within the micro-well plate wells are not 
subject to adverse jarring or adverse acceleration. Top-left-side corner flat 910 
provides orientation reference for tray 901a. Rounded top-right-side corner 912 is 
similar to the other two corners on bottom-rail 906d of tray 901a. Four second-cut- 
down areas 911a - 91 Id provide robotic access to allow for unstacking and stacking 
of the trays. 

FIG. 79 is an enlarged partial perspective front-side view of tray 901a showing pocket 
907a formed between top rail 906b, interior rail 908a, left rail 906a and right rail 906c 
and having bottom support 913. The preferred internal dimensions of the pockets 
between the enclosing rails are approximately 90 mm by 142 mm Planar bottom 
support 913 has rectangular recess forming a bottom-plate-shelf 914. Opening 915 
provides for structural flatness of the 901a in the molding process. Four robotic 
access cut-outs 916a - 916d, allow a robotic gripper to extend the horizontal plane of 
bottom-plate-shelf 914 and grasp a micro-well plate within pocket 907a. For 
example, robotic grippers on work cell gantry 905 can grab a micro-well plate in tray 
901a and place it on either of the two proteomic crystal verification and inspection 
systems 100, shown in FIG. 43. Bight tapered guide pillars 917a - 917h provide for 
centering and guiding a micro-well plate into 901a when the micro-well plate is 
placed into the tray. The tapered guide pillars 917a - 917h have their side facing the 
interior of the pocket sloped at preferably 5 degrees away from normal to the planar 
bottom-support 913 such that as the micro-well plate is placed within the pocket, the 
tapers help to center the plate as it is set onto the recessed bottom-plate shelf 914. The 
guiding features of pillars 917a - 917h work together with the access cutouts 916a - 
916d to allow easy robotic storing and retrieval of the plates, even though micro-well 
plates may have slightly different dimensions from manufacturer to manufacture. 

FIG. 80 shows an enlarged cross-sectional view of section A-A of FIG. 79 showing 
the planar bottom support 913 with its rectangular recess forming a bottom-plate-shelf 
914. A second shelf 918b is formed in the rim of the rails of the tray such that groove 
918a formed in the back-side of tray 901a will locate and properly nest into the 
second shelf 91 8b of a second tray when two or more trays are stacked together, one 
on top of the other. 
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FIG. 81 shows a bottom view of tray 901a showing the six open areas, corner flat 910, 
rounded corner 912, robotic access cut-outs 916a - 916x, and groove 918a for nesting 
of trays. 

FIG. 82 is an enlarged partial perspective view of tray 901a holding a 96 well micro- 
well plate 919 and positioned to illustrate the visibility of bar code 220 for a bar-code 
identifier for the plate. A properly positioned external bar-code reader may read the 
barcode of the micro-well plate while the plate is contained within the pocket in the 
tray. The cut-down area 909 in the top-rail of the tray provides access for a robotic 
tray gripper. The top-left-side comer flat 910 provides orientation reference for the 
tray. The rounded top-right-side comer 912 is similar to the other two corners on a 
bottom side of the tray. 



Although the above-preferred embodiments have been described with specificity, 
.persons skilled in this art will recognize that many changes to the specific 
embodiments disclosed above could be made without departing from the spirit of the 
invention. For example, although in the above preferred embodiments automated 
storage and retrieval system 900 was shown as being used in conjunction with the 
automated receiving machine identified as proteomic crystal verification and 
inspection system 100, automated storage and retrieval system 900 could be used to 
store and retrieve a variety of subject matter for a variety of purposes other than 
proteomic crystal verification and inspection. For example, in another preferred 
embodiment, the two protein crystal verification and inspection systems 100 could be 
replaced with two other automated receiving machines. In one preferred embodiment, 
the two protein crystal verification and inspection systems 100 are replaced with two 
automated micro-well plate filling machines similar to the one disclosed in 
Applicant's U.S. Patent Application Serial No. 09/702,164. In this preferred 
embodiment, storage gantry 903 removes trays containing empty micro-well plates 
from storage rack 932 and delivers the trays to work cell gantry 905. Work cell 
gantry 905 sequentially transfers the trays to the automated micro-well plate filling 
machines in a fashion similar to that described above in reference to the transfer of 
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trays to the protein crystal verification and inspection systems 100. After the micro- 
well plates have been filled, storage gantry 903 delivers the trays to storage rack 932 
for storage. Also, although access drawer 902 was discussed in detail in the above 
preferred embodiments, it would also be possible to utilize other types of access 
devices other than the access drawer described for depositing and removing trays. For 
example, the above invention could be configured so that an operator can insert trays 
into automated storage and retrieval system 900 by placing them on a flat table. 
Storage gantry 903 would then remove them from the flat table and store the trays in 
storage rack 932. Also, although the above preferred embodiments discussed the 
utilization of two protein crystal verification and inspection systems 100, it would 
also be possible to utilize just one of the two protein crystal verification and 
inspection system 100. Also, although the above preferred embodiments showed 
preferred tray 901a being utilized with automated storage and retrieval system 900, it 
could also be used with a variety of other automated machines that have robotic 
grippers. Also, although the above preferred embodiments discussed work cell gantry 
905 moving micro-well plates, work cell gantry 905 could also move other types of 
subject matter stored in trays other than micro-well plates. For example, work cell 
gantry 905 could also move pharmaceuticals, disposable health care products such as 
catheters and syringes, and biological compounds in micro-well plates or in individual 
bottles in storage trays, or the tray itself could be an individual micro-well plate 
handled directly by the gantry. In addition, though the present invention is described 
in terms of 6 micro-well plates to a tray, any number of micro-well plates could be 
handled with the appropriately sized tray, from one to 24 or more. Also, with regards 
to the proteomic crystal verification and inspection system, although the above 
preferred embodiments specifically describe an indexing device in which linear 
actuators 115, 150, and 160 operate in conjunction to sequentially position protein 
crystals under cameras 155 and 135, there are a variety of other types of robotic 
indexing devices that could also be utilized for the same purpose. For example, an 
indexing device could be built in which the plurality of micro-well plates are kept in a 
stationary position. The camera lens would be attached to an indexing device that is 
preferably capable of unrestricted movement in the horizontal plane. The camera lens 
would be moved sequentially from micro-well to micro-well in the horizontal plane. 
Once in position over a micro-well, the lens could be raised or lowered in the vertical 
direction to achieve proper zoom and focus. In another embodiment, an indexing 
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device could be built in which cameras 155 and 135 are kept stationary with respect to 
horizontal movement. In this embodiment, the plurality of micro-well plates would 
be preferably placed on a positioning platform that is capable of unrestricted 
movement in the horizontal plane. In this fashion, the positioning platform could be 
moved so that each micro-well is sequentially positioned underneath the appropriate 
camera. As with the previous embodiment, once in position over a micro-well, the 
lens could be raised or lowered in the vertical direction to achieve proper zoom and 
focus. Also, although the first preferred embodiment discussed inspecting crystals 
grown by the hanging drop method, other crystals grown utilizing other methods 
could be inspected with equal effectiveness. For example, FIG. 12 shows protein 
crystal growth as a result of aqueous drop in oil protein crystallization. Cameras 135 
and 155 focus on the crystals in drop 362. Also, although the above preferred 
embodiments discussed in detail how the present invention is utilized for inspecting 
protein crystals inside drops of liquid, the present invention could also be utilized to 
inspect other types of microscopic specimens. For example, the present invention 
. could be utilized to inspect typical micro-well micro titer plate reactions wherein the 
quality of the reaction can be judged by the amount and wavelength of fluorescence 
emitted by the specimen by configuring the system with appropriate light sources, 
filters, and sensitive cameras as is typical for fluorescence detection. . Also, although 
the above preferred embodiments disclosed the utilization of two cameras 135 and 
155, it would also be possible to have just one camera that is capable of zooming out 
so that it can focus on the entire well and zooming in so that it can focus on the drop 
of liquid containing the crystal. In addition, although an area CCD camera is shown, a 
linear CCD camera combined with moving of the micro-well plate would also work in 
the present invention. Also, in another preferred embodiment the detents 510 and 520 
can be simply spring loaded and not controlled by the computer 105. Also, the 
invention is taught with a light panel light source, and or a bulb, likewise LED light 
sources and laser fight sources could also be used with the present invention. 
Although the system is shown that only moves the micro-well plates in one axis and 
the camera in the other two axes, the invention could likewise be practiced with either 
the micro-wells moving in two orthogonal axes (such as X and Y) while the camera 
moves only in the Z-axis or the motion of all three axes be done with the camera 
system, wherein the micro-well plates are stationary and the system moves above 
them. These other variations of system design could also require rearrangement of the 



43 



WO 03/064047 PCT/US03/02107 

light source or multiple light sources. Also, other filter types may be substituted for 
second filter 354. For example, a linearly polarized filter would be very effective. 
Also, although the above preferred embodiments disclosed specific types of cameras 
135 and 155, other CCD cameras may be used in the present invention with less 
resolution or with greater resolution and still practice the present invention. For 
example, cameras of 2,000 by 2,000 pixels and even 4,000 by 4000 pixels are 
commercially available from several vendors. When digitizing these alternative 
cameras, the digitized image would have the corresponding resolution of the camera. 
Also, one may practice this invention and digitize to greater gray-scale accuracy than 
8-bit and gain advantage if the camera supports the greater bit depth, for example if 
the camera were cooled to reduce image noise. Therefore, the attached claims and 
their legal equivalents should determine the scope of the invention. 
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We claim: 

1) An automated storage and retrieval device for trays holding subject matter, 
comprising: 

A) an access device for the insertion and removal of a plurality of trays, 

B) a storage rack for storing said plurality of trays, 

C) at least one automated receiving machine, 

D) a work cell gantry, for moving said subject matter to and from said at least one 
automated receiving machine, and 

E) a storage gantry for moving said plurality of trays between said access device, 
said storage rack and said work cell gantry, 

F) at least one computer system programmed to control said storage gantry and 
said work cell gantry. 

2) The automated storage and retrieval device as in Claim 1, wherein said access 
device is an access drawer. 

3) The automated storage and retrieval device as in Claim 1, wherein said at least 
one automated receiving machine is a device for inspecting and classifying a 
plurality of microscopic crystals. 

4) The automated storage and retrieval device as in Claim 1, wherein said at least 
one automated receiving machine comprises: 

A) at least one camera, 

B) an indexing device for sequentially placing said microscopic crystals in 
camera- view of said at least one camera, and 

C) at least one control computer programmed to control said indexing device and 
said at least one camera, wherein said at least one control computer is 
programmed to receive from said at least one camera images of said plurality 
of microscopic crystals, wherein said at least one control computer is 
programmed to classify said plurality of microscopic crystals. 

5) The automated storage and retrieval device as in Claim 4, wherein said at least 
one control computer automatically classifies said plurality of microscopic 
crystals after receiving said images. 
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6) The automated storage and retrieval device as in Claim 1, wherein said at least 
one automated receiving machine is an automated micro-well plate filling 
machine. 

7) The automated storage and retrieval device as in Claim 1, wherein said at least 
one automated receiving machine comprises: 

A) a micro-well plate filling assembly, comprising: * 

1 . an indexing device, and 

2. a fill mechanism in communication with a media source and positioned to 
insert portions of said media into the empty micro-well plates, and 

B) an automatic control unit programmed to cause said indexing device to move 
empty micro-well plates adjacent to said fill mechanism, and to cause said fill 
mechanism to inject media from said media source into wells in the micro- 
well plates. 

8) The automated storage and retrieval device as in Claim 1, wherein said subject 
matter is at least one micro-well plate. 

9) The automated storage and retrieval device as in Claim 8, wherein said at least 
one micro-well plate comprises a bar code, wherein said automated storage and 
retrieval device further comprises at least one bar code reader in communication 
with said at least one computer system. 

10) The automated storage and retrieval device as in Claim 1, wherein said plurality of 
trays holds at least one micro-well plate, wherein said storage gantry and said 
work cell gantry each comprise at least one robotic gripper, wherein said plurality 
of trays comprises: 

A) at least one cut-down access area for said at least one robotic gripper, 

B) a comer flat for tray orientation, and 

C) a plurality of tapered guide pillars for guiding said at least one micro-well 
plate into said plurality of trays. 

1 1) A tray for holding at least one micro-well plate, wherein said tray is handled by at 
least one robotic gripper, wherein said plurality of trays comprises: 
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A) at least one cut-down access area for said at least one robotic gripper, 

B) a corner flat for tray orientation, and 

C) a plurality of tapered guide pillars for guiding said at least one micro-well 
plate into said tray. 

12) A method for automatically storing and retrieving a plurality of trays, wherein 
said plurality of trays holds at least one micro-well plate, comprising the steps of: 

A) inserting said plurality of trays onto an access device, 

B) transferring via said storage gantry said plurality of trays to a storage rack, 

G) transferring via said storage gantry said plurality of trays to a work cell gantry, 

H) transferring via said work cell gantry said at least one micro-well plate from 
said plurality of trays to at least one automated receiving machine, 

I) transferring via said work cell gantry said at least one micro-well plate from 
said at least one automated receiving machine to said plurality of trays, 

J) transferring via said storage gantry said plurality of trays to said access device, 
K) removing said plurality of trays from said access device, and 
L) controlling the movement of said storage gantry and said work cell gantry via 
at least one programmed computer system. 

13) The method as in Claim 12, wherein said access device is an access drawer. 

14) The method as in Claim 12, wherein said at least one automated receiving 
machine is a device for inspecting and classifying a plurality of microscopic 
crystals. 

15) The method as in Claim 12, wherein said at least one automated receiving 
machine comprises: 

A) at least one camera, 

B) an indexing device for sequentially placing said microscopic crystals in 
camera-view of said at least one camera, and 

C) at least one control computer programmed to control said indexing device and 
said at least one camera, wherein said at least one control computer is 
programmed to receive from said at least one camera images of said plurality 
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of microscopic crystals, wherein said at least one control computer is 
programmed to classify said plurality of microscopic crystals. 



16) The method as in Claim 15, wherein said at least one control computer 
automatically classifies said plurality of microscopic crystals after receiving said 
images. , 

17) The method as in Claim 12, wherein said at least one automated receiving 
machine is an automated micro-well plate filling machine, 

18) The method as in Claim 12, wherein said at least one automated receiving 
machine comprises: 

A) a micro-well plate filling assembly, comprising: 

1 . an indexing device, and 

2. a fill mechanism in communication with a media source and positioned to 
insert portions of said media into the empty micro-well plates, and 

B) an automatic control unit programmed to cause said indexing device to move 
empty micro- well plates adjacent to said fill mechanism, and to cause said fill 

• mechanism to inject media from said media source into wells in the micro- 
well plates. 

19) The method as in Claim 12, wherein said plurality of trays holds at least one 
micro-well plate. 

20) The method as in Claim 19, wherein said at least one micro-well plate comprises a 
bar code, wherein said automated storage and retrieval device further comprises at 
least one bar code reader in communication with said at least one computer 
system. 

21) The method as in Claim 12, wherein said plurality of trays holds at least one 
micro-well plate, wherein said storage gantry and said work cell gantry each 
comprise at least one robotic gripper, wherein said plurality of trays comprises: 

A) at least one cut-down access area for said at least one robotic gripper, 

B) a corner flat for tray orientation, and 
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C) a plurality of tapered guide pillars for guiding said at least one micro-well 
plate into said plurality of trays. 

22) An automated storage and retrieval device for trays holding subject matter, 
comprising: 

A) an access means for the insertion and removal of a plurality of trays, 

B) a storage rack means for storing said plurality of trays, 

C) at least one automated receiving machine means, 

D) a work cell gantry means for moving said subject matter to and from said at 
least one automated receiving machine, 

E) a storage gantry means for moving said plurality of trays between said access 
device, said storage rack and said work cell gantry, and 

F) at least one computer system means programmed to control said storage 
gantry means and said work cell gantry means. 

23) The automated storage and retrieval device as in Claim 22, wherein said access 
means is an access drawer. 

24) The automated storage and retrieval device as in Claim 22, wherein said at least 
one automated receiving machine means is a device for inspecting and classifying 
a plurality of microscopic crystals. 

25) The automated storage and retrieval device as in Claim 22, wherein said at least 
one automated receiving machine means comprises: 

A) at least one camera, 

B) an indexing device for sequentially placing said microscopic crystals in 
camera-view of said at least one camera, and 

C) at least one control computer programmed to control said indexing device and 
said at least one camera, wherein said at least one control computer is 
programmed to receive from said at least one camera images of said plurality 
of microscopic crystals, wherein said at least one control computer is 
programmed to classify said plurality of microscopic crystals. 
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26) The automated storage and retrieval device as in Claim 25, wherein said at least 
one control computer automatically classifies said plurality of microscopic 
crystals after receiving said images. 

27) The automated storage and retrieval device as in Claim 22, wherein said at least 
one automated receiving machine means is an automated micro-well plate filling 
machine. 

28) The automated storage and retrieval device as in Claim 22, wherein said at least 
one automated receiving machine means comprises: 

A) a micro-well plate filling assembly, comprising: 

1 . an indexing device, and 

2. a fill mechanism in communication with a media source and positioned to 
insert portions of said media into the empty micro-well plates, and 

B) an automatic control unit programmed to cause said indexing device to move 
empty micro-well plates adjacent to said fill mechanism, and to cause said fill 
mechanism to inject media from said media source into wells in the micro- 
well plates. 

29) The automated storage and retrieval device as in Claim 22, wherein said subject 
matter at least one micro-well plate. 

30) The automated storage and retrieval device as in Claim 29, wherein said at least 
one micro-well plate comprises a bar code, wherein said automated storage and 
retrieval device further comprises at least one bar code reader in communication 
with said at least one computer system means. 

31) The automated storage and retrieval device as in Claim 22, wherein said plurality 
of trays holds at least one micro-well plate, wherein said storage gantry means and 
said work cell gantry means each comprise at least one robotic gripper, wherein 
said plurality of trays comprises: 

A) at least one cut-down access area for said at least one robotic gripper, 

B) a corner flat for tray orientation, and 
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C) a plurality of tapered guide pillars for guiding said at least one micro-well 
plate into said plurality of trays. 

32) A tray for holding at least one micro-well plate, wherein said tray is handled by at 
least one robotic gripper, wherein said plurality of trays comprises: 

A) at least one cut-down access area means for said at least one robotic gripper, 

B) a corner flat means for tray orientation, and 

C) a plurality of tapered guide pillar means for guiding said at least one micro- 
well plate into said tray. 

33) A method for growing protein crystals, comprising the steps of: 

A) inserting drops of protein solution into a plurality of micro-well plates, 

B) placing said plurality of micro-well plates onto a plurality of trays, 

C) inserting said plurality of trays onto an access device, 

D) transferring via said storage gantry said plurality of trays to a storage rack, 

E) transferring via said storage gantry said plurality of trays to a work cell gantry, 

F) transferring via said work cell gantry said plurality of micro-well plates from 
said plurality of trays to an automated machine for inspecting and classifying a 
plurality of microscopic crystals, 

G) analyzing via said automated machine said drops of protein solution, 

H) transferring via said work cell gantry said plurality of micro-well plates from 
said at least one automated receiving machine to said plurality of trays, 

I) transferring via said storage gantry said plurality of trays to said access device, 
J) removing said plurality of trays from said access device, and 

K) controlling the movement of said storage gantry and said work cell gantry via 
at least one programmed computer system. 

34) Protein crystals grown by a method comprising the steps of: 

A) inserting drops of protein solution into a plurality of micro-well plates, 

B) placing said plurality of micro-well-plates onto a plurality of trays, 

C) inserting said plurality of trays onto an access device, 

D) transferring via said storage gantry said plurality of trays to a storage rack, 

E) transferring via said storage gantry said plurality of trays to a work cell gantry, 
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F) transferring via said work cell gantry said plurality of micro-well plates from 
said plurality of trays to an automated machine for inspecting and classifying a 
plurality of microscopic crystals, 

G) analyzing via said automated machine said drops of protein solution, 

H) transferring via said work cell gantry said plurality of micro-well plates from 
said at least one automated receiving machine to said plurality of trays, 

I) transferring via said storage gantry said plurality of trays to said access device, 
J) removing said plurality of trays from said access device, and 

K) controlling the movement of said storage gantry and said work cell gantry via 
at least one programmed computer system. 
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.StartZ value 
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MoVe Z axis to Z_Position(N) = Slart_Z value 
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N = N+1 



w Acquire and ' 
Store Image(N) 
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Move Z axis to 
Z_Position(N)- Z_Posltion{N) + 
Z_Step_Size 
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Extract lmage(N' 
focus Value F(N) 
Subroutine 
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320 



$t_Focus = F(N) 
Best Z = N 



No 



No. 
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Best_Z = 
Number_of_?_Sfices/2 
Focus_Error=1 



4-Yes 
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Set Bestjmage « lmage(Best_Z) 
Set BestJmage_Z c Z_Position(BesLZ) 
Seta Pointer Jo3esUmage 
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Extract Image(N) 
focus value F(N) 
Initialization 
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PolnterJo_StoredJmage 



408 
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Convolve Imaged) with Horz_SobelFBtertofind 
Horizontal Edges creating lmage(NLH 
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414. 



Convolve Image(N) with Vert_SobeI Filter to find 
Vertical Edges creating tmage(N)_V 



\ 



SUM lmage{NLH + lmage(N)_V to create 
lmage{NLFocus 
Truncate values to range of 0 to 255 



Calculate Variance F(N) In 
lmage(NLFocus over the 
desired window 



▼ / 

Q RETURN* ^ 
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' XJMndcwjDenter 
Y Window_Center 
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Start 
Specimen Auto 
t Score & Classify J 
Subroutine 



442 
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Initialization 



448 



450 



PointerJoJJesUroaga / 
Type_of.Classlflcat!on I 



446 



Calculate Average_Gray_Background normalized for well! type by 
multiplying by Micro-well normal izer value 



Calculate Average J3ray_Cfassify_Window 



454 



Calculate DIff Image 1 
If ABS[PbceI(x,y) - Plxel(x+Diff Sep.y+Dfff Sep)] > RagJThresh 
THEN 

DlfrjmageJ w .PIxel(x,y)= Setl 
ELSE 

Dlffjmage 1 Pixelfx.y) = 0 



456 



Calculate DIff Image.? 
If ABS[Plxelfc+DifLSep I y) . Pixelfcy+DifLSep)] > 
RagJThresh 
THEN 

Dlffjmage^ Plxel(x ( y)= Sett 
ELSE 

DlffJmage_2_Pixe!(x«y) = 0 
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460 



Calculate Classify Image 
If Dlffjmage 1 Pixel(xV)» Setl 
\ OR 

DiffJmageJ2J>lxel(x,y)= 5 Setl 
THEN 

ClassHyJmage Pixel{x t y) ? Set2 
ELSE 

ClassJfyJmage_PixeI(x t y) = 0 
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Calculate thB number_PlxeIs_Set2 
Calculate Total Pixels in Window 



LGray = Av6fageJ3rayJ3lassiryj/Vind^ 
Score_Flag = Number_Pixels_Set2/TotaLPixelsJn_Window 



464 



CALL 

Classify (Score_Gray, Score Rag) subroutine 
FIG 34a step 468 
Type_of_Classlfication as parameter 



XJ3ackGround_Center 
Y_BackGround Center 
X_BackGround Width 
Y_BackGround_Height 
Micro-well type Identifier 
Micro-well normer value 
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X Classify Center 
Y_Classlfy~Center 
X ClasshyJA/idth 
Y.CIassity^Helght 
DIff Sep 
Setl * 
Set2 

RagJThresh 



RETURN 
to FIG 36 
step 852 
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480 



Calculate CLEAR 
If Score F1ag< Clear_FlagJ-T 
AND 

Score Gray > Clear_Gray_GT 
THEN 
CLASS = 0 




484 



Calculate Light Precipitation 
If Score_Flag < Lgt_PrecipJ 5 lagJT 
AND 

Score Rag > Lgt Predp_FlagjGT 
AND 

Score Gray > Lgt Precip_Gray_GT 
THEN 
CLASS = 1 



Calculate Heavy Precipitation 
If Score JHag < Heavy_Precip_Flag_LT 
AND 

Score_F1ag > Heavy_Predp_Rag_GT 
AND 

Score Gray < Heavy_Precip_GrayJLT 
THEN 
CLASS = 2 
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Calculate Ugly Precipitation 
If Score Flag < Ugly_Precip_Flag_LT 
AND 

Score_Flag > Ugly_Precip_F1ag_GT 
AND 

Score Gray < Ugly_PrecIp_Gray_LT 
THEN 
CLASS = 3 
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\ Clear_Gray_GT I 
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Calculate Micro_Crystals 
If ScoraLFlag > Micro Cry FlagLGT 
AND 

ScoreJSray < Micro_Cry Gray_LT 
THEN 
CLASS = 8 



Calculate Crystal 
If Score JHag < Crystal_FlagJ.T 
AND 

Score_Rag > CrystaLFlag GT 
AND 

Score Gray > CrystaLGray GT 
THEN 
CLASS = 9 



Calculate GRAINY Precipation 
If CLASS = 8 
AND 

Score_Flag > Grainy_Flag GT 
AND 

ScoreJBray > Grainy_Gray GT 
THEN 
CLASS = 7 
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Micro_Cry_FlagjGT 
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CrystaLRagJLT 
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FIG. 34b 



WO 03/064047 



20/50 



PCT/US03/02107 



725 



727 




726 



\ Calculate a first set of Additional Image 

Features in Classifyjmage 
CALL Subroutine MVTJToolsJB!obJ\nafysis 
with pointer to Window in Classifyjmage 
Get 
numjound 
and for each found GET 
area(m), heightfm), width(m), perimeter(m) 
location JC(m), Iocation_Y(m) 



X_ClassHy_CentBr 
Y Classify Center 
X~ClassHy Width 
YjClassify_Heigrit 
PointerJTojCiassifyJrnag 

mvLUob_analysis_j>arams 
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728 
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m = 1 
num_spherolite = 0 
spheroDte(m) » 0 



730 



Calculate Blob_Ratios 
Circle Like HW(m) = heightfm)/width(rn) 
Circle_UkeJVHW(m) - area(my(height(m)*Width(m)) 



732 



No 



Calculate .Spherolite 
If CLASS » 8 OR 9 
AND 

Circle Like HW(m)<CfrcleJJIce„HW - .Upper 
AND 

Circle Like HW(m) >CirdeJJkeJMJj>wer 
AND 

Circle Uke J\HW(m) < Circle UkeJ\HW_Upper 
AND 

Circle UkeJ\HW(m)>arcle.UkeJ^HW_Lower 
THEN 

nurnjspherolite* num_spherolite +1 
m = m+1 
6pherolite(m) = 1 
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if nurn_Spherolite > 0 then CLASS = 6 




FIG. 34c 
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Calculate a second set Additional Image 
Features fn Bestjmage 
Using the results from the previous 
MVT_Tools__Blob ^Analysis on the 
Class ffyjmage 
m = 0 
num j>hase_sep = 0 
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(Averag8_Gray_Background 

Poln!er_To_Best_lmagB 

nun^found 

area(m), hetght(m), 

wldth(m), perimetari[m) 

locatton_X(m) 

tocaHon w Y{m) 

num_spheroIite 

spherolita(m) 





758 



Usi/ig locatton_X(m) l !ocation H Y(m), 1/2(height(m)) t 1/2(width(m) 
calculate the average gray value within this reduce window for each 
spherolite previously found and set phase_sep_Gray(m) = to the 
found value 
then Normalize it by calculating 
phase_sep_Gray(m) =phase_sep_Gray(m)/ 
Average GrayJ3ackground 
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f phase sep_Gray > phase sep Gray GT 
THEN 
CLASS = 4 
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Analyze previously measured features from FIG 34c to 
further separate CLASS 9 into sub-classes 9.0 through 
9.9 
setm = 0 

First test if CLASS = 9 f If not then goto step 778 
if true then 
CLASS = 9.0 
if num found *1 then goto step 768 
ifnotgoto769 



-YES- 




766 



Uverage_Gray__Backgr 
ound 

FointerJTo_BestJmage 
numjbund 
areaTm), height(m), 
wldth(m), perimeter(m) 
locationJ((m) 
location_Y(in) 
num_spherofite 
spherottte(m) 
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771 



m» 1 

HW Ratio = Height(m) / Width 
\ " (m) 
if HW_Ratio > needle_HW_GT 

" or 
HW Ratio < needle_HW J-T 
"the n CLASS* 9.2 



770 




needle HWJ3T 
needle_HW.LT 




If HWJtetio > plate JHWJ3T 
or 

HW Ratio < plate_HW.LT 
Ihen CLASS » 9.4 
THEN goto 773 ELSE goto step 778 
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772 



plate HWjST 
plateJJWJT 



I— YES- 



Using facation X(m), location JT(m) ( 1/2(height(m)) , 1/2(wldth(m) .) 
calculate the average gray value within this reduce window for the Blob 
previously found and set Chunk_Gray(m) * to the found value 

then Normalize it by calculating 
Chunk J3ray(m) ==ChunkJ3ray(m) /Average_Gray_Background 
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\ If Chunk Gray < Chunk.GrayJLT 
THEN 
CLASS a 9.6 



^-775 
- (chunk_Gray_LT ( 



\ If area(m) > Chunk.50.GT 
THEN 
CLASS - 9.8 



^~ 777 
-(chur^_50„GT ( 



If area(m) > gorgeous_GT 
THEN 
CLASS = 9.9 
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-|gorgeous_GT ^ 
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FIG. 35a 



Return to FIG 33 ^ 
step 466 
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For m » 0 to numjbund 
Calculate for each blob 
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.Average J3ray_Background 

Polnter_To_BesUmage 

numjbund 

area(m), height(m), wldth(m), 

perimeter(m) 

tocation_X(m) 

tocation_Y(m) 

num_spherolite 

spherolile(m) 



m = m + 1 
HW Ratio = Helghtfm) / Width 
(m) 

If HW.Ratio > needle JHW_GT 
or 

HW_Ratlo < needleJHWLT 
then CLASS = 9.1 



794 




790 



needla HW_GT 
needle HWLT 



If HW_Ratto > plate_HW_GT 
or 

HW Ratio < plate HW_LT 
Then CLASS = 9.3 
THEN goto 798 ELSE goto step 812 
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plate HW.GT 
plate_HWJ.T 



YES 



798 



Using location_X(m), tocation^m), 1/2(height(m)), 1/2{width(m) ) 
calculate the average gray value within this reduce window for the Blob 
previously found and set Chunk_Gray(m) = to the found value 

then Normalize It by calculating 
Chunk_Gray(m) =Chunk_Gray(m) /AverageJ3rayJ3ackground 
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\ If Chunk Gray .< Chunk Gray_.LT 
THEN 
CLASS = 9.5 
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If area(m) > Chunk_50_GT 
THEN 
CLASS = 9.7 




^ChunKjGray_LT ( 



^-808 
-|chunk_50_GT ( 
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FIG. 35b 



Return to FIG 33 
step 466 
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Initialization Parameters 
{ Inspection lists 
Type_af_Classification 



I 1 




846 



Move to micro-well within 
selected micro-well plate 




-NO- 



849 



YES* 



acquire image and locate 




droplet 






-856 j 



Move to droplet within 
micro-well In micro-well 
plate 



acquire droplet high- 
resolution image 




NO 



850 



CALL Specimen Auto Score & 
Classify Subroutine 
FIG 33 step 440 
Type_of_Classif!cation as 
" parameter 
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FIG. 39 
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